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ABSTRACT 


A  channel  is  defined  to  be  a  triple  S  ■  {X,  u(.|x),  (Y,  Y’i) 
where  X  and  Y  are  abstract  sets,  Y'  is  a  o-algebra  of  subsets  of 
Y,  and  w(.[x)  is  a  probability  measure  on  (Y,  Y')  for  each  xcX. 

X  and  Y  are  usually  assumed  to  be  either  subsets  of  the  real 
numbers  or  subsets  of  Euclidean  n-space.  The  channel  has  addi* 
tive  noise,  u,  if  u(A[x)  -  w(A-x)  for  all  xeX,  AcY\  For  a 
given  channel  S  and  a  given  real  nusber  0  <  X  <  1  a  X-code  of 
length  n  for  S  is  a  set  of  pairs  (x^,  Dp  ...  (xn,  Dn)  where 
xicX,  D.cY»,  Din  Dj  -  a  if  i  ^  j  and  vCD.jxp  ?  1-X  for  i  -  1, 

2,  ...,  n.  The  supremum  of  the  nonempty  set  of  integers  N  such 
that  S  admits  a  X-code  of  length  N  is  denoted  by  N(S,  X). 

Let  Q*(S)  be  the  set  of  all  probability  measures  defined  on 
X,  For  ncQ*(S).  Let  y*  be  the  measure  defined  on  (Y,  Y')  by 

YW(B)  ■  /  u(B|x)dw 

for  all  BcY ' ,  For  xcX,  let  f^Cyjx)  ■  du(. [ x) /dyn  (the  Radon- 
Nikodym  derivative).  The  capacity .  C,  of  the  channel  S  is 
defined  to  be  sup  {C(*) |w cQ*(S) ,  support  of  w  finite},  where 

CC")  -  //  log  f  "(y  i  x)  v(dy|x)n(.dx). 

The  following  improvement  of  Fano's  theorem  is  proved  in 


in 


Chapter  II, 


Theorem:  Let  S  be  a  channel  with  capacity  C,  and  let 


0  <  X  <  1  be  given.  Ihen  for  any  n>0,  0<t<l 

Cl- XJ  log  N(Sn,  X)  <  nC  ♦  log  (ilk. 

tA 

In  Chapter  III  the  connected  X-code  is  defined  and  investi¬ 
gated.  This  code,  {x^t  is  defined  to  be  a  X-code  where 

each  is  a  O-connected  set;  U  is  a  topology  on  Y.  The 

supremum  of  the  set  {N|S  admits  a  connected  X-code  of  length  N} 
is  denoted  by  N(S,  X,  U) .  Most  of  the  results  of  this  chapter 
are  for  a  channel  of  type  A  -  a  channel  with  additive  noise  u 
which  is  absolutely  continuous  with  respect  to  LebeSgue  measure, 
y;  X  is  a  closed  interval;  Y  is  the  real  nusbers;  and,  U  is  the 
usual  topology  for  the  reals. 

Let  f  ■  du/dy,  the  Radon-Nikodym  derivative,  f  is  a  bell 
function  if  there  exists  yQ  such  that  f  is  increasing  for  all 
y  ~  yQ  and  f  is  decreasing  for  all  y  £  y  .  In  Chapter  III  two 
conditions  for  N(S,  X).  ■  N(S,  X,  II)  are  obtained  when  S  is  a 
channel  of  type  A  and  f  is  a  bell  function.  One  of  these  condi¬ 
tions  is  necessary;  the  other  is  sufficient. 

A  technique  for  delineating  those  measures  which  cannot 
affect  the  value  of  N(S,  X,  U).  is  investigated  in  Chapter  IV. 


IV 


Sufficient  conditions  for  a  channel  to  have  finite  capacity 
are  investigated  in  Chapter  V.  The  main  result  is: 

Theorem:  Let  S  be  a  channel  with  additive  noise  y 
which  is  absolutely  continuous  with  respect  to  Lebesgue  meas¬ 
ure,  y.  If  there  exists  a  choice  for  the  Radon-Nikodym  deriva¬ 
tive,  f  ■  dy/dy,  such  that  /  g(y)dY  <  •  where  gC/1  ■ 
sup  {f(y[x)|xcX}  then  the  capacity  of  S  is  finite. 
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CHAPTER  I 


INTRODUCTION  AND  PRELIMINARIES  !  ' 

Most  of  the  terminology  and  notations  which  will  be  used 
herein  are  standard;  however,  to  remove  any  possible  aa&iguity, 
much  of  it  will  be  explained.  For  a  set  A,  acA  means  that  a  is 
a  member  of  A;  the  complement  of  A  will  be  written  AC.  It  will 
sometimes  be  convenient  to  refeT  to  a  member  of  A  as  a  point  of 
A.  All  logarithms  will  be  to  the  nse  e.  The  convention  will 
be  adopted  throughout  to  define  the  expression  0  log  0  to  be 
equal  to  0.  Integrals  will  always  be  in  the  sense  of  Lebesgue 
integration.  The  meaning  of  unexplained  terminology  or  notation 
of  integration  theory  or  measure  theory  will  be  that  of  Halmos  [4], 
and  the  meaning  of  any  unexplained  probability  theory  terminology 
or  notation  will  be  that  of  Loeve  [6], 

Information  theory  is  one  of  the  youngest  branches  of  applied 
probability  theory.  Its  conception  can,  with  certainty,  be  con¬ 
sidered  to  be  the  appearance  in  1948  of  the  now  classical  work  of 
Shannon  [7].  From  the  very  begining,  information  theory  presented 
to  mathematicians  a  whole  new  set  of  problems,  including  some  very 
difficult  ones.  It  is  quite  natural  that  early  investigators, 
including  Shannon,  whose  basic  goal  was  to  obtain  practical  results. 
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were  not  able  to  give  enough  attention  to  these  matheaatical 
difficulties.  Consequently  at  many  points  of  their  investiga¬ 
tions,  they  were  convened  either  to  be  satisfied  with  reasor- 
ing  of  an  inconclusive  nature  or  to  limit  the  set  of  objects 
studied. 

Investigations  with  the  aim  of  setting  information  theory 
on  a  solid  matheaatical  basis  have  begun  to  appear  in  recent 
years.  However,  in  most  of  these  endeavors  finiteness  conditions 
have  been  placed  on  certain  sets  in  order  to  establish  the  desired 
results. 

One  of  the  most  important  entities  considered  in  the  mathe¬ 
matical  study  of  information  theory  is  the  concept  of  a  channel. 

A  channel  is  defined  to  be  the  triple  S  ■  {X,  u(.|x),  (Y,  Y')} 
where  X  is  an  arbitrary  set,  (Y,  Y*)  is  a  measurable  space,  and 
u(.|x)  is  a  probability  measure  on  (Y,  Y')  for  each  xeX.  The 
set  X  is  usually  referred  to  as  the  input  or  inj>utj5jj>ace;  the  set 
Y  is  referred  to  as  the  output  or  output  space.  If  both  X  and  Y 
contain  only  a  finite  number  of  elements,  the  channel  is  said  to 
be  discrete;  if  X  has  a  finite  nuaber  of  elements  but  Y  is  infinite 
(either  countable  or  uncountable),  S  is  called  semi- continuous  (the 
term  "semi- continuous"  is  of  engineering  origin). 


In  most  of  the  literature  it  is  assumed  that  the  channel 


is  either  discrete  or  semi- continuous .  Such  restrictions  will 
not  Be  made  in  this  dissertation;  it  will  be  assumed  throughout 
that  X  «nd  Y  are  arbitrary  sets  unless  there  is  a  specific  state¬ 
ment  to  the  contrary. 

A  channel  operates  as  follows.  The  existence  of  a  sender 
and  receiver  is  assumed.  The  function  of  the  sender  is  to  choose 
a  menber  cf  X  and  transmit  it.  Since  it  is  not  necessary  to  have 
a  precise  definition  of  the  term  transmit,  a  somewhat  heuristic 
explanation  is  given.  Transmission  consists  of  choosing  a  point 
xcX  and  associating  with  x  a  point  ycY  which,  in  general,  depends 
on  x.  The  points  x  and  y  will  be  called  the  transmitted  syafcol 
(transmitted  signal,  syufeol  sent,  etc.)  and  the  received  syefcol 
(received  signal,  received  point,  etc.),  respectively.  If  a 
particular  transmitted  syafcol  always  results  in  the  same  received 
syvfcol,  the  transmission  may  be  considered  as  a  function  T:  X  Y. 
The  more  interesting  case  is  that  in  which  a  given  transmitted 
signal  does  not  always  result  in  the  same  received  sysbol.  In 
this  case  the  function  T  must  be  considered  as  a  function  of  x 
and  another  variable,  called  the  noise.  The  received  variable  is 
considered  to  be  a  chance  variable,  i.e.,  specific  occurrences 
are  governed  by  probability.  The  only  property  of  the  transmission 
known  by  the  sender  and  receiver  is  that  for  each  x  sent  the 
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probability  that  the  received  syrtol  is  a  aerter  of  AeY*  is 
u(A[x).  The  raceiver  is  capable  of  scanning  through  any  finite 
class  of  sets  of  Y'  and  determining  which,  if  any,  contains  the 
received  symbol.  After  making  this  determination,  the  receiver 
then  tries  to  decide,  based  upon  some  type  of  logical  analysis, 
which  mealier  of  X  was  actually  transmitted. 

The  technique  which  is  usually  employed  by  the  receiver  to 
decide  which  point  was  transmitted  involves  a  predetermined 
decision  scheme  which  is  known  to  both  the  sender  and  the  receiver. 
There  are,  of  course,  many  ways  by  which  this  decision  sheme  can 
be  defined.  The  one  which  has  become  a  standard  in  studies  of 
information  theory  is  as  follows:  Let  0  5  X  <  1  be  given;  let 
(Xj,  Dj),  ...,  (xn,  Dn)  be  Barters  of  X  «  Y'  having  the  properties 
that  Dj  ■  9  if  i  #  j,  and  such  that  u(D^|x^)  £  l-\.  The 

sender  and  receiver  then  agree  to  consider  only  those  raerters 
Xj,  X2,  ...»  xneX.  If  x^  is  transmitted,  the  receiver  scans  through 
the  sets  D^,  ...,  Dn  and  determines  which,  if  any,  contains  the  re¬ 
ceived  syrtol.  If  the  received  syrtol  is  in  Dffl,  the  receiver  con¬ 
cludes  that  x  was  transmitted:  if  none  of  these  sets  contain  the 
01 

received  symbol,  any  decision  may  be  made.  The  receiver's  conclusion 
will  be  correct  with  probability  l  1-A. 

1.1  Definition:  A  set  of  pairs  {(x^,  D^,  ...,  (xn,  Dn) }  hav¬ 
ing  the  properties  described  above  is  called  a  A- code  of  length  n. 


A  quantity  which  will  be  of  considerable  inport ance  later, 
in  fact  that  subject  of  most  of  the  i^sortant  theorems  in  informa¬ 
tion  theory,  is  defined  below. 

1.2  Definition:  Lot  S  be  a  channel.  Given  0  5  X  <  1,  let 
N(S,  X)  denote  the  supreain  (sup.)  of  the  non-eapty  set  of  inte¬ 
gers  N  such  that  S  admits  a  X-code  of  length  N. 

Given  n  channels  S  -  {X  ,  u  (.|x  ),  (Y  ,  Y’)},  m  «  1,  2, 

m  mm  m  mm 

...»  n,  one  can  form  the  product  channel.  •  Sj  *  ...  *  Sn  in 

a  natural  way.  In  fact  this  channel  is  defined  by  »  (X^, 

( .  | u)  ,  (YCnl,  Y(n)')}  where  XC"J  -  Xj  *  ...  *  Xn;  (YCnl, 

Y^n**)  is  the  product  of  the  measurable  spaces  (Y^,  Y^.m-1.  2. 
....  n,  as  defined  by  Halmos  (4);  and  if  u«X,  i.e.,  u  •  (x^,  ..., 

xn)  with  *neXB),  then  (.|u)  denotes  the  product  probability 

measure  on  Y^  '  defined  by  (Bj  *  ...  *  Bn)  ■  uCB^Xj)  .... 

y(Bn|xn)  where  BjCY’,  B  cY^. 

1.3  Definition;  The  channel  defined  above  is  called  a 

memoryless  channel  of  length  n.  If  Sj  ■  Sj  »  ...  *  SR  ■  S,  one 
writes  S*“n^  ■  Sn  and  calls  Sn  the  memoryless  channel  of  length  n 
generated  by  S.  Any  channel  may  be  regarded  as  a  memoryless  channel 
of  length  1. 


1.4  Remark:  The  definitions  of  N(S(n\  X)  end  N(Sn,  X) 
follows  ioMdiately  fro*  definitions  1.2  and  1.3. 

Suppose  there  are  N  distinct  points  in  X  which  the  sender 

wishes  to  transmit  in  such  a  manner  that,  for  a  predetermined 

0  *  X  <  1,  the  probability  that  the  receiver  will  wrongly  deduce 

which  (.oint  was  sent  is  -  X.  The  channel  S  cannot  necessarily 

perforq  this  function  if  N  >  N(S,  X).  However,  if  for  fixed  X, 

N(Sn,  X)  beoones  unbounded  with  n,  the  problem  can  be  solved  by 

n0 

choosing  an  nQ  such  that  N(S  ,  X)  >  N,  establishing  a  one-to-one 
correspondence  between  the  N  points  and  a  properly  chosen  set  of 
N  members  of  XR®,  and  using  the  channel  s”0. 

This  problem  and  two  equivalent  (according  to  Holfowitt  (ll]l 
versions  are  listed  below: 

Fora  I  :  Given  N  and  X,  how  small  an  n  will  suffice? 

Form  II  :  Given  n  and  X,  how  big  an  N  can  be  achieved? 

Fora  III:  Given  n  and  N,  how  small  a  X  can  be  achieved? 

A  companion  problem  to  the  above  problem  (call  it  the  first) 
is  the  (second)  problem  of  constructing  a  code  to  implement  the 
answer  to  the  first  problem.  In  fact,  it  might  be  reasonably 
thought  that  the  first  problem  oould  not  be  solved  without  a 
solution  of  the  second.  This  is  not  the  case,  and,  at  present, 
existing  knowledge  about  the  first  problem  considerably  exceeds 


the  existing  knowledge  about  the  second  problem 


One  of  the  nain  objectives  of  this  dissertation  is  to  define 
a  technique  for  coding  (called  a  connected  X-code)  and  to  inveeti* 
gate  what  conditions  oust  be  placed  on  the  channel  so  that  such  a 
code  will  provide  a  solution  to  the  second  problem. 

The  solution  of  the  first  problea  is  usually  referred  to  as 
the  coding  theorem  and  its  converse.  Precise  statements  of  these 
theorems  must  be  reserved  for  later,  since  they  involve  terminology 
which  has  not  yet  been  introduced 

The  other  main  objective  of  this  dissertation  is  to  obtain  an 
inprovement  of  the  known  results  for  the  converse  of  the  coding 
theorem  for  the  general  channel.  It  will  be  seen  later  that  this 
aaounts  to  obtaining  an  improvement  of  the  known  results  for  an 
upper  bound  for  N(Sn,  X)  where  S  is  a  general  channel  (the  channel 
whew  both  the  input  and  the  output  are  arbitrary  sets). 

Before  continuing  toward  these  objectives,  seme  basic  note* 
tion,  definitions,  and  theorems  will  be  listed  for  fiiture  refer* 
ence.  Since  information  theory  employs  many  of  the  tools  of  pro- 
bability  theory  and  measure  and  integration  theory  this  list  a 
fortiori  contains  results  from  these  disciplines. 


I  I 

In  addition  to  probability  atatenenta  about  the  received 
aymbol,  many  of  the  important  reaulta  entail  probabillatic 
atatenenta  about  the  transmitted  aymbol.  The  following  develop- 
menta  will  indicate  the  importance  and  application  of  auch 
atatementa  aa  well  aa  provide  a  rigorous  foundation  for  their 
formulation. 

l.S  Definition:  Let  S'  -  (X,  u(.|x),  (Y,  Y')}  be  a  given 
channel.  The  set  Q*  -  Q*(S')  is  defined  to  be  the  collection 
of  all  entities  > 

a  -  {X,  X',  x} 
o  o 

of  the  following  kind:  The  set  Xq  (called  the  support  of  w) 
is  any  fixed  subset  of  X,  X^  is  a  o  -algebra  of  subsets  of  Xq 
containing  all  sets  of  the  form 

{x|xcXq,  g(B|x)  <  a.BtY' ,a  real} 
i.e.,  such  that  the  function  u(B|x)  is  measurable  in  x  for 
fixed  BcY'.  Finally  x  is  a  probability  measure  on  the  o- 
algebra  X\ 

The  apparent  ambiguity  in  letting  the  Greek  letter  x 
both  represent  and  be  a  member  of  the  entity  {Xq,  X^,  x) 
will  cause  no  confusion  in  usage  and  will  allow  for  simpli¬ 
city  in  notation. 


9 


1.6  Definition:  Let  (Y,  Y • )  be  a  measurable  space,  and 
let  y  and  y  be  measures  on  Y* .  Then  y  is  absolutely  continuous 
with  respect  to  y  (written  y  <<  y)  if  y(A)  «  0  implies  y(A)  »  0; 
U  is  singular  with  respect  to  y(yly)  if  there  exists  a  set  AeY* 
such  that  y(A)  »  0  and  yCA0)  •  0. 

The  following  well-known  theorem  is  stated  for  complete¬ 
ness. 

1.7  Lebesgpe  Decomposition  Theorem:  Let  u  and  y  be  meas¬ 
ures  defined  on  the  measurable  space  (Y(  Y'){  then  y  can  be 
written  uniquely  as  the  sum  of  two  measures  Uj  and  u2  where 

Uj  «  Y  and  y^y. 

1.8  Remark:  Any  set  D  such  that  y(D)  ■  y(Y)  is  called 

a  support  of  y,  This  will  be  written  D  ■  spt  u.  If  ujjr  D  can 
be  chosen  such  that  y(D)  •  0.  Whenever  y«  y^+y2  U1  K<  7 

and  u2Jjr  a  support,  D,  can  always  be  chosen  for  y^  i»l,  2 
such  that  Uj(D)  ■  0  jyi.  Throughout  this  dissertation  such  a 
choice  for  a  support  will  always  be  implied. 

1.9  Remark:  Let  y  and  y  be  defined  as  above  and  let 
y  -  Uj+yj  where  yj  «  y  and 

urable  set  D  such  that  y(D)  ■  0  and  yjCD0)  *  0.  The  set  D  is 
called  a  singular  set  of  y  with  respect  to  y.  Let  g(y)  ■  dy^/dy 


y2J_y  •  Then  there  exists  a  meas- 
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(the  R'ion-Nikodym  derivative),  then  g(y)  i*  a  finite-valued, 
non-negative,  real-valued,  y- measurable  function,  unique  up  to 
a  set  of  Y*nuun  0,  such  that 

MjCBI  •  /  g(y)dY,  for  BcY* 

By  definition  of  D,  one  is  allowed  to  assuae  that 

0  <  g(y)  <  -  if  ycDc, 

g(y)  ■  -  if  ycD. 

Throughout  this  dissertation  such  a  determination  of  Radon- 
Nikodym  derivatives  will  always  be  chosen. 

The  preceding  developments  provide  a  suitable  background 
for  the  definition  of  a  new  set  of  probability  measures  on 
(Y,  Y’).  Let  iteQ*.  For  BcY’  define 

Y*(B)  -  /  y(B(x)ir(dx) . 

It  is  easy  to  see  that  y*  is  a  probability  measure  on  (Y,  Y') 
for  each  *cQ*. 

Let  f'tylx)  ■  dyj/dy*  where  u^C.jx)  is  the  absolutely  con¬ 
tinuous  component  of  u(.|x)  with  respect  to  y*.  loeve  [6]  has 
shown  that  f*(ylx)  can  be  chosen  such  that  it  is  jointly  meas¬ 
urable  in  xeXQ,  yeY'  relative  to  the  a  -  algebra  X^  *  Y' .  By 
definition  of  the  set  D  in  Remark  1.9,  it  is  clear  that  f^CyJx) 


can  (and  Mill)  be  chosen  so  that  it  also  satisfies  this  joint 
measurability  condition  in  addition  to  the  requirement  spec¬ 
ified  in  the  remark. 

The  measure  yw  and  the  functions  f*(y|x)  Mill  hom  be 
used  to  define  an  important  characteristic  of  the  channel, 
the  Capacity. 

Let  «cQ*  be  axbitrary  but  fixed.  Define 

C(")  ■  //  log  £WCy l x)  y(dy | x)  »(dx) , 

*  //  fW(ylx)  £WCy 1 x)  Y*(dy) *(dx) 

if  U(.|x)  <<  y*  for  almost  all  [*]  xcX.  Otherwise,  define 
C(ir)  -  ♦  •, 

If  the  support  of  *  is  countable,  the  folloMing  equiva¬ 
lent  definition  of  C(ir)  will  be  convenient.  If  u(.|x)  «  y*, 
define  C(x|n)  ■  /  log  f*(y|x)  u(dy|x).  It  is  observed  that, 
in  this  case,  u(.[x)  «  yv  for  alirost  all  [n]  x  in  the  support 
of  it.  Hence, 

C(TT)  -  l  TT(X)  C(x|»). 

XCSptTT 

1.10  Definition:  Let  Q  «  {»eQ*|  support  of  w  is  finite). 
Let  C  ■  sup  {C(w)  | ttcQ} .  The  quantity  C  is  called  the  capacity 


of  the  channel. 


The  following  important  and  somewhat  surprising  result  in 
due  to  Keapernan  [5], 

1.11  Theorem:  Let  C*  ■  sup  (C(w) | wcQ* } .  Then  C*  ■  C. 

The  coding  theorem  and  its  converse  can  now  be  stated. 

The  following  statements  of  these  theorems  are  those  of 
Wolfowitx  [11]. 

1.12  Theorem:  (The  coding  Theorem  ) :  Let  0  <  X  <  1  be 
given.  Then  there  exists  a  positive  constant  K  such  that  for 
any  n  >  0 

ncs",  x)  ;  .nC  *  . 

1.13  Theorem;  (The  strong  Converse).  Let  0  <  X  <  1  and 
c  >  0  be  given.  Then  for  any  n  sufficiently  large 

n(s".  x)  ;  ."<c  *  c). 

1.14  Theorem:  (The  weak  Converse).  Given  0  <  X  <  1. 
Then  for  all  n  >  0 

( 1- X)  log  N(Sn,  X)  *  nC  ♦  log  2. 

Theorem  1.12  was  conjectured  for  the  discrete  channel  by 
Shannon  (7]  in  1948.  The  first  proof  was  given  by  Feinstein 
[1]  in  1954.  Essentially  different  proofs  were  given  in  1957 
by  Shannon  [8]  and  Wolfowitz  [9],  Shannon  also  conjectured 


theorem  1.14  for  the  discrete  channel.  The  strong  converse  is 
due  to  Wolfowitz  [9]. 

Nolfowitz  [10]  has  shown  that  theorem  1.12  is  true  for  a 
semi continuous  channel  by  approximating  the  semicontinuous 
channel  by  a  discrete  channel.  The  proof  of  theorem  1.13  for 
the  semicontinuous  channel  is  also  due  to  Wolfowitz  [10].  The 
following  stronger  version  is  due  to  Kenperman  [S] . 

1.15  Theorem:  Let  0  <  X  <  1  be  given.  Then  for  any 
semicontinuous  memoryless  channel  S  there  exists  a  constant 
K  >  0  such  that  for  any  n  >  0 

N(S\  i)  _<  .nC  *  K/ir  . 

Theorem  1.14  is  due  to  Fano  [2]  who  proved  it  for  the 
general  channel.  An  essentially  different  proof  of  this 
theorem  has  been  given  by  Kenperman  [5],  A  different  expres¬ 
sion  for  the  right-hand  side  of  the  formula  given  in  theorem 
1.14  will  be  obtained  in  Chapter  II.  For  small  A  this  will 
give  a  much  better  result  [in  the  since  of  a  smaller  ipper 
bound)  than  the  one  given  in  1.14,  The  following  well  known 
theorem  which  will  be  needed  for  the  proof  is  listed  for 
reference. 
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1.16. Theorem:  Let  m  ■  1,  n  be  arbitrary  channel* 

of  capacity  Cn.  Then  the  capacity  of  the  product  channel 

i*  C.  ♦  ...  ♦  C  . 
l  n 


CHAPTER  II 


AN  IMPROVEMENT  OF  FANG’S  THEOREM 

It  is  not  unrsual  in  the  field  of  mathematics  for  one  to 
conjecture  an  extension  of  a  known  result  to  a  note  general 
setting  without  being  able  to  obtain  a  proof.  Sonatinas  this 
conjecture  renains  an  open  problen  for  many  years.  This  is 
the  current  status  of  both  the  coding  theoren  and  the  strong 
converse  for  the  general  charnel. 

In  the  foraulaticn  of  a  theoren  the  primary  abjective  of 
which  is  to  obtain  an  upper  bound  for  sons  quantity,  one 
usually  attempts  to  establish  as  vnell  an  upper  bound  as  poss¬ 
ible.  The  author  is  unaware  of  any  theoren  which  gives  abetter 
upper  bound  for  N(S,  A)  for  a  general  channel  than  theoren  1.14, 
Fano's  theoren.  The  theoren  presented  below  gives  an  iaprove- 
ment,  for  small  A,  of  Fano's  result.  The  proof  of  theorem  2.1 

4 

is  a  modification  of  the  proof  of  1.14  given  by  Kenpexaan  [5J. 

2.1  Theorem:  Let  S  ■  (X,  w(.|x),(Y,  Y ' )  }  be  a  given 
channel.  Then  given  0  <  A  <  1,  0  <  t  <  1,  one  has,  for  each 
positive  n 

(1-A)  log  N(Sn,  A)  <  nC  ♦  log 

t 

where  C  is  the  capacity  of  S. 
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Proof: 

It  follows  from  theorem  1.16  that  the  capacity  of  the 
channel  Sn  is  nC;  therefore,  it  suffices  to  prove  th«t 

(1-X)  log  N(S,  X)  *  C  ♦  log  (-i-AA. 

t 

Let  N  be  any  positive  integer  such  that  N  i  N(b  ,  X).  Then 

N 

there  exists  a  X-code  of  length  N;  call  it  {(x. ,  j.  f-et 

A  ■  (x^lCx.,  is  a  merfjer  of  this  X-code).  (This  notation 
could  be  singilified  by  assuming  that  x^'s  were  distinct  menbers 

N 

of  X).  For  e.ich  xcA  let  N(x)  •  l  l  1,  and  let  ir(x)  ■  N(x)/N. 

i»l  X^Xj 

It  is  easy  to  see  that  >  is  a  probability  measure  on  X  with 
finite  support.  (More  precisely  tcQ(S).) 

For  each  BcY'  define 

y*CB)  •  l  y(&lx)*(x). 
xcA 

As  has  been  pointed  out  y*  is  a  probability  measure  on  (Y ,  Y ' ) 
and  w(.|x)  «  y*  for  all  xcA.  For  *(x)  >  Q  let 

f* (y l x)  •  du(. |x)/dyn. 

Now,  by  definition, 

l  it(x)  /  fn(ytx)  log  fwCy i *)  yffC«fy)  •  c(*l  1  c. 
xcA 
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Cbcerve  that 


,  N  , 

N  I  ^(ylx.i  -  N*1  l  N(x)  f*(y | x) 
i- 1  xcA 


«  l  *(x)  f*  (y  |  x)  -  1. 
xcA 


Siailary , 


t  N 

N'1  I  /  f'Cyk)  log  f\ylx.)Y1,Cdyl 

i-1  1  1 


■  I  "(*)  /  *"(ylx}  log  f*(ylx)Y*(«bO  f  c. 

xcA 


A  new  set  of  functions  is  defined  on  Y  as  follows.  For 
each  1  *  i  *  N,  define 


N  if  ycDi 

h-Cyl  ■ 

t  if  yeD? 

N 

Then  £  h.  (y)  *  N(l*t)  for  each  y. 
i-1  1 


The  desired  results  will  now  be  obtained  by  analyzing  the 

terns  of  the  following  equation: 

.  N  h.Cyl 

N”1  /  I  f^Cylx.l  log— -i - Y^Cdy) 

i-1  1  f  Cy|x.i 


.  N 

♦  N‘*  l  I  f*(y  |x.)  log  fw(yix.)YW(dy) 

i-1  1  1 

t  N 

•  N r*  i  /  f’Cykl  log  h  (/)/(.<*>. 

i-1  11 

For  rtfmnc*  this  Mill  bs  called  equation  (*1 .  . 

For  each  0  .<  i  -<  N 

/  f  *(y  l  x^)  log  hi(y)y,T(dy)  -  /  log  h^y)  ;:(<&' (Xj I 

•  wCD-ix^  log  N  ♦  uCD'J  |xiJ.  log  t. 

I  (1  -  \y  log  N  ♦  A  log  t. 

.  N 

Therefore,  N"  l  J  f  (y|x.)  log  h. (y)Y*(dy) 
i-1  1  1 

l  Cl  -  *1  log  N  ♦  A  log  t. 

The  function  log  Z  is  concave;  hence, 

N  h.  (yj  .  N 

N"1  l  f  (ylx.J  log— i - «  log  N’1  l  h.  Cyl  <  log  (1  ♦  U. 

i-1  1  f  (ylx^  i-1 
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7V/4"** 

'l**r"* 


.  N  h. (y) 

Therefore,  N_1  /  l  f*(y|x.)  log  -—2 - y*(dy)  -  log  (1  ♦  t). 

i-1  1  f (y Ix±> 

Inserting  these  inequalities  into  equation  (•) ,  one  obtains 
log  (1  ♦  t)  ♦  C  i  (1  «  X)  log  N  ♦  X  log  t. 


which  is  equivalent  to 

(1  -  XI  log  N  <  C  ♦  log 

t 

Since  N  f  N(S,  X)  was  arbitrary,  the  theorem  is  proved. 

2 . 2  Remark ;  Given  0  <  X  <  1/2  let  f(t)  »  (1  ♦  t)/t* 

for  t  c(0,  1) ;  f(t)  ■  ♦  •  if  t  ^(0,  1) .  Then  f  has  a  adniwua  at 
t  ■  X/C 1  -  X),  The  function  g(X)  »  X_*(l  -  X)*”*  is  an  increas¬ 
ing  function  of  X  and  g(y)  <  2. 

2.3  Corollary:  Let  S  be  a  given  channel.  Then  for 

0  <  X  <  1  one  has,  for  each  positive  n,  (1  -  X)  log  N(Sn,  X)  * 
nC  -  X  log  X  ♦  log( 1  ♦  X).  If  0  <  X  <  1/2  then 

(1  -  X)  log  N(Sn,  X)  *  nC  -  X  log  X  -(1  -  X)  log  (1-  X), 


•$ 

r 


£ 

i 


& 

f 


“i 

f 
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CHAPTER  III 


CONNECTED  A-CODES 

Although  for  a  given  channel.  S,  the  value  of  N(S»  A) 
may  be  known  for  each  0<A<  1,  the  actual  construction  of  a 
A-code  of  length  N(S.  A)  may  be  quite  difficult.  In  addi¬ 
tion.  for  practical  reasons,  it  may  be  desirable  to  place 
various  restrictions  upon  the  entitles  of  the  code.  Such 
restrictions  may,  of  course,  preclude  the  possibility  of 
attaining  a  code  of  maximal  length.  In  this  chapter,  a 
A-code  with  a  specified  restriction  is  defined  and  analyzed. 

In  the  operation  of  a  channel,  one  of  the  functions  of 

the  receiver  is  to  scan  through  the  sets  of  a  given  A- 
N 

code  {x^,  0^}^^  and  determine  which  of  these,  if  any,  contains 
the  received  signal.  This  determination  may  be  quite  difficult 
the  degree  of  the  difficulty  depending  on  the  nature  of  the 
sets  (Recall  that  the  only  restriction  placed  on  the 

D^'s  is  that  they  may  be  members  of  Y'.) 

3.1  Definition:  Let  U  be  a  topology  on  Y.  A  (/-connected 

A-code  is  a  A-code  (x, ,  D.),  ...»  (x  ,  D  )  where  D,  is  a  U- 
—  A  i  n  n  l 

connected  set  for  each  i-1,  2,  . . . ,  n. 

3.2  Definition:  For  a  given  channel  and  a  fixed  real 
number  0<A  <1,  let  N(S,  A,  U)  denote  the  supremum  of  the 


non-empty  set  of  integeis  N  such  that  S  admits  a  U- connected 
X-code  of  length  N. 

It  is  clear  that  N(S,  X,  U)  1  N( S ,  X)  for  each  0  <  X  <  1 
and  any  topology  U,  It  is  also  clear  that  equality  need  not 
hold  unless  some  restrictions  are  inposed  upon  the  channel. 

It  appears  intuitively  clear  that  these  restrictions  should 
be  placed  on  the  set  of  probability  measures  (u( . |x) |x*X)  and 
must  include  restrictions  upon  the  supports  of  these  measures. 
Although  it  would  be  desirable  to  let  X  and  Y  be  arbitrary 
sets  and  U  be  any  topology  on  Y ,  such  generality  leads  to  com¬ 
plicated  and  unwiedly  analysis  and  yields  very  few  results.  In 
this  chapter  some  restrictions  will  be  placed  upon  the  input 
space  and  the  output  space  as  well  as  the  set  of  probability 
measures.  These  restrictions  will  allow  for  almost  all  of  the 
practical  physical  situations  one  might  expect  to  encounter. 

The  following  example  indicates  some  of  the  restrictions 
which  must  be  plaoed  upon  this  set  of  probability  measures. 

3,3  Example:  Let  X  =  {0 ,  1} ;  let  Y  =  [0 ,  1];  let  Y*  be 
the  Borel  sets  on  [0,  1];  and  let  y  be  Lebesgue  measure.  The 
measures  y(.|l)  and  y(.|o)  are  defined  as  follows:  For  BcY' 

u(B|0)  =  /  C-4x+2)dy  +  /  (4x-2)dy, 


define 


m (B 1 1)  ■  /  4xdy  ♦  /  (-4x*4)dy, 

B1  B2 

when  Bj  -  BH  tO,  1/2]  and  Bj  -  B^  [1/2,  1]. 

Let  U  be  the  usual  topology  for  [0,  1].  Since  X  contains 
only  2  elements  N(S,  X)  ~  2  for  all  0  <  X  <  1/2.  Observe  that 
w([0,  1/4]  U  (3/4,  1]|0)  -  3/4  .  u([l/4,  3/4] |l].  Hence 
N(S,  1/4)  i  2.  It  is  easy  to  see  that  if  is  any  interval 
contained  in  Y  such  that  w(Io|0)  £  3/4  then  y(1q)  >  1/2;  also  if 
1^  is  any  interval  contained  in  Y  such  that  w(ljl)  -  3/4  then 
Y(Ij)  l  1/2.  It  follows  that  N(S,  1/4,  U)  -  1. 

In  the  exanple  above,  one  is  able  to  obtain  a  longer  n on- 
connected  code  becase  of  the  nature  of  the  Radon-Nikodyn 
derivative  of  u(.|0)  with  respect  to  Lebesgue  Measure.  It  will 
be  observed  that  if  one  chooses  0  <  X  <  1  and  considers  any  con¬ 
nected  set  A  such  that  w(A|0)  *  1-X ,  then  there  exists  a  set  B 
such  that  u(B|0)  ■  1-X  and  y(B)  <  y(A). 

Although  it  was  not  difficult  to  show  that  N(S,  1/4)  »  2  while 
N(S,  1/4,  U)  ■  1,  it  is  easy  to  see  that  this  problen  could  rapidly 
become  difficult  as  the  nunber  of  points  in  X  is  increased.  Given 
xcX,  let  f(y|x)  ■  du(.|x)/dy.  The  relationship  between  N(S,  X)  and 
NCS,  X,  U)  for  arbitrary  X  is  very  difficult,  in  fact,  almost 
impossible,  to  determine  when  X  is  infinite,  unless  one  requires 


som  typo  of  uni fo rarity  among  tboso  functions*  Such  a  restric¬ 
tion,  to  bo  rigorous  1/  defined  later,  will  be  placed  upon  the 
channels  investigated  in  this  chapter. 

The  following  auxiliary  results  will  be  needed  later. 
Throughout  the  renainder  of  this  chapter  R  will  denote  the  real 
mahers.  S  will  be  the  Borel  sets,  y  will  be  Lebesgue  Mature, 
and  U  will  be  the  usual  topology  for  the  reals.  If  Y  is  a 
subset  of  R  then  the  o-algebra  Y*  will  be  the  Borel  sets  on  Y. 

3.4  Theorem:  Let  u  be  a  totally  finite  Mature  of  (R,  8). 
Let  <| <  be  any  mbsu re  defined  an  (R,  8).  Let  a  >  v(R) ,  let 
0  <  X  *  a  be  given,  and  let  A  ■  {Ac8:  y(A)  l  a-X).  Then  A  con¬ 
tains  a  Maher  of  minimal  V- measure . 

Proof: 

Let  u  “  *  1*2  “^ere  U1  ~  ♦  and  Let 

Aq  *  spt  Wj*  If  w(A0)  -  the  theorem  is  trivially  true. 

Suppose  not.  Let  8  »  a-X-y(Ao).  Let  f(y)  ■  dy^/d^.  Let 

E  ■  {y:  f(y)  >  :),  Observe  z.  <  z.  implies  E  O  E  .  Note 

Z1  z2 

that  E  is  a  immeasurable  set  for  each  zcR.  Let  I(z)  »  /  f(y)  d*. 

Ez 

Then  I(z)  is  non-negative,  monotone  non- increasing  and  left- 
con  ti  uous.  Let  zQ  -  inf  {z:  l(z)  5  8).  Observe  that 


24 


Not  if  B C  E.  and  CC  E*  ,  than  /  f(y)d*  >  /  f(yld*  >  0 
xo  xo  C  B 

i splits  KO  >  *(B>.  [IfycC  then  f(yl  <  *Q  while  for  ycB, 

Hy\  -  »J.  Hence  if  I(zQ)  -  6  then  Ez  U  Aq  is  clearly  a  set 

0 

of  minimal  \hmeasure  such  that  u(Ez  U  A^  2  a- A,  Suppose 

0 

I(z)  >  8.  Let  E'  ■  {y:  fCy)  >  iQ)  and  E"  ■  (y :  f(y)  »  zQ). 

If  y(E')  ■  8  then  AJJE'  is  the  required  set.  If  u(E')  <  8  then 
w(E")  >  8  -  u(E')  >  o.  Let  E,H  be  a  subset  of  E"  of  ^-measure 
(8  -  w(E'l)/zo.  Then  u(E'U  E"')  -  6  and  E •  U E"  has  minimal 
measure.  Hence  A  -  A  UE'llE'"  is  a  set  such  that  v(A)  2  o-A 
and  such  that  *CB)  2  *(A)  for  any  set  B  such  w(B)  2  a- A. 

3.5  Remark:  Let  A  be  any  set  of  minimal  ^measure  such 
that  u(A>  2  a- A.  Let  B  ■  spt  v2  and  let  yQ  *  inf  {yl«1C-  m,  y) 

>  »-A  -  u2(B)}.  Let  f(y)  •  dw^/d*.  If  fCyj)  2  f(y2>  for  almost 
*11  [#]  y^  <  yQ  *  y2  then  A  can  be  chosen  to  be  A  ■  BU(-",  yQ). 

The  proof  of  the  following  two  Corollaries  comes  imnediately 
from  the  proof  of  theorem  3.4. 

3.6  Corollary:  Let  f  be  a  non-negative  real-valued  Lebesgue 

measurable  finction.  For  BcB  and  0  i  a  <  j  f(yjdy  there  exists  a 

B 

set  ACB  of  minimal  y- measure  such  that  /  fdr  2  o. 

A 

3.7  Corollary;  Let  v  be  a  totally  finite  measure  defined 
on  (R,  8).  Let  a  -  uCR)*  If  V  «  Y,  then  there  exists  AeB  such 
that  u(A)  «  a-A  and  y(B)  2  y(A)  for  any  Bc8  such  that  y(B)  2  <*-A. 


In  several  of  the  theorem  developed  in  this  chapter,  it 
will  be  hypothesized  that  wC«l*)  «  y  for  all  xtX.  The  follow¬ 
ing  the  ores  gives  a  partial  justification  for  such  requireaents. 

3.8  Theorea:  Let  v  be  a  totally  finite  asasure  on  (R,  81. 
Suppose  that  frr  each  0  2  X  *  y(R)  there  exists  A.c8  of  ainiaal 

i  ^ 

Y-aeasure  such  that  y(A^)  *  y(R}-X  and  A^  is  connected.  Let 
V  ■  Uj  ♦  Uj  where  «  y  and  UjIt*  ^en  t^er®  exists  a  support 

of  u2  which  contains  at  aost  one  point  (jspt  w2l  -  1). 

Proof: 

Suppose  [apt  w2l  l  2.  Let  A  ■  {a[a  ■  apt  w2  and 
YCAl  *  o).  Then  given  AcA ,  A  is  not  connected.  Let  X  -  y(R)  - 
U2CR1  and  let  be  any  set  of  ainiaal  yneasuxe  such  that 
uCA^l  ■  wCR)  "  A.  Observe  that,  given  AcA,  y(A}  -  y(R)  .  X. 

Hence  yCA^)  ■  0.  Therefore  v(Ap  •  that 

A^cA;  hence  is  not  connected. 

3.9  Lemma:  Let  f:  R  ■*  R  be  a.e.  continuous  and  non¬ 
negative  such  that  0  <  /  fdy  ■  o  <  «*.  Suppose  there  exists 

R 

x^  <  x2  <  Xj  such  that  f  is  continuous  at  x^  i«l,  2,  3,  and 

f(x1)  >  fCxjl  <  fCx3l.  Then  there  exists  X  >  o  such  that,  if 

A  is  any  set  of  ainiaal  y- measure  such  that  /  fdy  2  a-X,  A  is 

A 


not  connected. 


Proofs 


2 6 


Ut  E  -  lx;  f(x\  *  1/2  ♦  ain  fC*3)3).  Ut 

B  ■  /  fUldy, 

E 

Ut  Ej  ■  E  (\  {xs  x  i  Xj), 

e2  *  E/V*:  *  -  *2}» 

and 

B,  •  /  f(*XdY  i  ■  1,  2. 

Ei 

Sinca  f  is  continuous  at  x^,  x2  and  x^,  it  is  easy  to  see 
that  0.(  +  o  i  8 ^  an^  ^  is  not  connected. 

Ut  X  >  o  be  chosen  such  that  81*B2*b-X>  max  (8j,  S2> . 

Now  xcE  implies  f(x)  >  l^tfCxj},  adn  (f(Xji  f(x3iJ}  and 
x«Ec  iaplies  that  f(x)  <  l/2{f(x2),  Bin  (f(Xj},  f(Xj)l).  Thus, 
if  A  is  any  set  of  mi  rival  y-measura  such  that  /  f(x>dy  -  o-X, 
then  y(A/TEcl  ■  0,  Moreover,  6^  >  o,  B2  >  o,  and  o-X  >  max 
(Bj,  821  iqjlies  AAE^  f  0,  AOE2  +  0.  Hence  A  is  not  connected. 

With  the  aid  of  Lemma  3.9,  one  can  characterise  those  a.e. 

continuous  sunaable  functions  f:  R  -*•  R  such  that  given 

o  <  X  <  /  fdy  there  exists  a  set  A.  of  minimal  y- measure  for 
R 

which  /  fdy  t  /  fdy  -  X,  which  is  connected. 

Aa  R 

3,10  Definition:  A  bell  function  is  any  function  f:  R  ■*  R 
such  that  there  exists  xQ  such  that  f  is  monotone  nondecreasing 
for  x  <  xQ  and  f  is  monotone  nonincreasing  for  x  >  xQ. 
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3.11  Theorem:  Let  f  be  a  non-negative  a.«.  continuous 

real-valued  function  defined  an  R  such  that  o  <  /  fdy  ■  a  < 

R 

Given  o  <  X  <  a  let  B  be  any  set  of  minimal  y- measure  such  that 

/  fdy  i  o-A,  Then  there  exists  for  every  X  a  connected  set  A 
B 

such  that  /  fdy  ■  /  fdy  and  y(A)  »  y(B)  iff  f  is  a  bell  function 
A  B 

a.e. 

Proof: 

Observe  that  if  f  is  not  a  bell  function  a.e.  then 
there  exists  points  of  continuity  Xj  <  x2  <  x^  such  that  f(Xj) 

>  f(x2)  <  f ( x3) .  Thus,  the  necessary  part  follows  immediately 
from  Lemma  3.9.  The  sufficient  part  will  be  proved  by  con¬ 
structing  the  set  A.  The  construction  is  similiar  to  that  used 

I 

in  3.4. 

Assume  f  is  a  bell  function.  Let  -  {x[f(xl  i  yh  Let 
I Cyi  ■  /  f(x)dy.  Let  y  ■  inf  {y:  I(y)  J  ►>},  Then  I(y  )  > 

P  o  o 

y 

o-X.  Let  E'  -  {x:  f(x)  >  yo>  and  E"  ■  {x:  f(x)  »  yo>.  Now 

it  is  clear  that  if  A^  and  A^  are  y-measurab  le  sets  such  that 

AjCEy  and  A^E^  and  o  <  yfA^  *  y(Ai)  then  /  fdy  >  /  fdy, 
o  0  A  |  Aj 

Thus  if  y(E"  )  ■  o  the  proof  . is  completed.  Suppose  y(E"  )  4  o. 

yo  yo 

Then,  E"  -  [a.,  b.]  U  [a,,  b,]  and  E’  -  (b  ,  a2)  where 
*o  1 0' 

bj  *  a2>  Clearly  one  can  choose  a  c  [a^  b ,  b  c  [a2,  b2J 
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such  that  B>  -a  ♦  b-a.]y  -  u *->-/  fdy.  Now  A.  ■  [a,  b]  is  the 

1  *  o  pV 

Vo 

required  interval. 

The  preceding  results  will  be  used  in  the  analysis  of  an 
important  special  channel  which  will  now  be  introduced. 

Let  S  be  a  given  channel.  If  anytiae  a  value  x  is  trans¬ 
mitted,  the  receiver  is  able  to  determine  from  the  received 
synbol  that  x  was  the  point  set,  then  the  channel  is  called 
noiseless.  Such  channels  rarely  occur  in  practice  and  are  cf 
little  mathematical  interest.  On  the  other  hand,  if  there  is  a 
definite  positive  probability  that  the  receiver's  decision  will 
be  wrong,  then  the  channel  is  called  noisy.  An  important  type 
of  noisey  channel  is  the  channel  with  additive  noise. 

3.12  Definition:  Let  S  ■  {X,  v(.|x),  (Y,  Y'}}  be  a  given 
channel.  S  has  additive  noise  if  there  exists  a  probability 
measure  w  on  (Y,  Y')  such  that  v(A|x)  »  u(A-x)  for  each  xcX, 

AcY'.  The  measure  u  is  called  the  noise. 

3.13  Remark:  In  order  to  assure  that  the  operations  indi¬ 
cated  in  3.12  are  well  defined,  it  will  be  assumed  throughout 
the  remainder  of  this  dissertation  that  if  S  is  a  channel  with 
additive  noise  then  (Y,  ♦)  is  a  group. 

3.14  Theorem?  Let  S  be  a  channel  with  additive  noise  u. 

If  u  <<  y ,  then  y(.|x)  <<  y  for  each  xcX  and  f(y[x)  ■  f(y-x) 
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when  f(y)  -  dw/dy  and  f(y[x)  ■  dw(.|x)/dy. 

Proof: 

By  the  Radon-Nikodym  theorem 

n(A|x)  -  w(A-x)  ■  /  f(y)dy  ■  /  f(y-x)dr. 

A-x  A 

Hence,  by  the  absolute  continuity  of  the  integral,  u(A|x)  «  y. 

Also,  by  the  Radon*Nikodym  theorem 

p(A|x)  •  /  f(y|x) dy. 

A 

Thenfore,  f(y[x)  ■  f(y-x)  for  almost  all  [y]  ycY. 

Now  it  is  clear  that  to  determine  a  necessary  and  sufficient 
condition  for  N(S,  A,  lij  ■  N(S,  A),  when  S  is  a  channel  with 
additive  noise,  one  need  only  investigate  a  single  measun,  the 
noise  w.  The  following  concept  will  play  an  important  role  in 
this  investigation. 

3,  IS  Definition:  An  interval  (a,  bj  is  left  adjusted  in 
an  interval  (c,  d)  iff  a  ■  c  and  b  *  d.  A  sequence  of  disjoint 
intervals  {(ai#  b^])™^  is  left  adjusted  in  an  interval  Cc,  d\ 
iff  (a^,  b^J  is  left  adjusted  in  (c,  d)  and  (a^,  b^J  is  left  ad> 
justed  in  (b^^,  d)  for  i  ■  2,  3,  ...,  n. 

3,16  Definition:  Let  |i  be  a  measure  defined  on  (R,  8). 

Let  0  -  A  <  1,  An  interval  (a,  bj  is  Cu»  A) •minimal  left  ad- 
justed  in  (c,  d)  iff  (a,  b]  is  minimal  left  adjusted  in  (c,  d), 
u(a,  b)  i  A#and  if  u(c,  b^  *  A  then  bj  i  b,  Let  be 


'  - 
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amuris  defined  on  (R,  3).  A  sequence  {(a^,  i# 

^wi^£«l*  adjusted  in  (c,  d)  iff  (a^,  bj  is 

Cu^»  X}-*iniaal  left  adjusted  in  (c ,  d)  and  (a.,  b^)  is  (Vj»  X)* 
■fniasl  left  adjusted  in  (b._^,  d)  for  i  ■  2,  3,  . ..,  n. 

3.17  Una:  Let  Y  ■  (c,  d).  If  u  is  s  totally  finite 
■assure  on  (Y,  Y'),  then  for  0  <  X  i  y(Y)  there  exists  an  inter* 
val  (a,  b]  which  is  (y,  X}-nininal  left  adjusted  in  Y. 

Proof: 

Let  A  ■  <b :  v(e,  b]  *  A}.  A  i  0  since  X  1  w(Y>,  Let 

b  -  inf  {b:  bsA).  Let  {b  }"  .  be  a  sequence  contained  in  A 
o  n  n-i 

which  converges  to  bQ.  It  say  be  assuaed  with  no  loss  of  gener¬ 
ality  that  {ba}^Bl,  is  a  decreasing  sequence.  Then  {(c, 
is  a  decreasing  sequence  of  intervals  such  that  v(c,  bR]  <  •  for 

m 

all  n  and  (c,  bQ]  »  f*\  (c,  .  Therefore  y(c,  bQ]  » 

lia  y(c,  b  ]  »  X.  New  it  is  clear  by  the  nature  of  bQ  that 
n-*-  n 

(c,  bQ]  is  (w,  X)-minimal  left  adjusted. 

3.18  Theorem:  Let  S  ■  {X,  u(.|x>,  (Y,  Y’JJbe  a  channel 
with  Y  connected.  Let  0  5  X  <  1  be  given.  Let  (Xj,  Dj) ,  . 

(xn,  0Q)  be  a  connected  X-code  of  length  N(S,  X,  U) .  Then  there 
exists  a  C( wC*  1^) }"Bl»  l-X)-ainiaal  left  adjusted  sequence. 

Proof: 

It  may  be  assumed  that  the  sequence  is  ordered; 


i.e.  if  on*  d*not*s  D.  »  (*i#  ]  than  1  <  j  iaplies  <  *y 

By  leasts  3.17,  th*r*  exists  *  (v(.|xj),  l-A)-siniaal  left  ad¬ 
justed  interval  in  Y.  Call  it  D^.  Sine*  is  (u(.lx^),  1-1)- 
niniaal  laft  adjusted,  it  is  clear  D2CY*D|;  henca, 
h(Y-DJ  |  Xj)  l  1-1.  Thus ,  again  by  lam  3.17,  there  exists  and 
intarval  which  is  (wC.lxj),  l-A)-siniaal  left  adjusted  in 
Y-DJ.  It  is  clear  that  proceeding  thus ly  on*  obtains  the  desired 
sequence  {Dp“  .. 

3. 19  Raeark :  In  leama  3.17  if  ti  «  y  than  x(c,  bQJ  “  A; 
hence  in  theorae  3. IS,  if  w(.|x)  «  Y  for  each  xcX  w(D^[x^)  * 

1-1  for  i  »  1,  2,  .  ,  n. 

3.20  Definition:  Let  S  •  {X,  uC*  l*)  »  (Y,  Y'}}  b*  a  channel 
with  additive  nois*  w  which  is  absolutely  continues*  with  respect 
to  y.  If  Y  is  the  real  nuufcers,  Y'  the  Borel  sets,  and  X  a 
closed  interval  contained  in  Y,  then  S  is  called  a  channel  of 
type  A. 

throughout  the  remainder  of  the  Chapter  the  e^haszs  will 
be  on  channels  of  type  A.  In  most  of  the  oialysis  it  will  also 
be  assumed  that  f  *  dy/dy  is  a  bell  function.  Tedmiques  for 
actually  constructing  connected  A- codes  of  length  !(S,  1,  IQ  will 
now  be  presented.  The  first  such  construction  is  for  an  arbi¬ 
trary  channel  of  type  A.  Since  y({y})  *  0  the  sets  will  be 
written  as  open  intervals. 


3.21  Construction:  Let  S  be  a  channel  of  Type  A.  Let 
0  2  X  <  1  be  given.  Let  X  •  [a,  bj.  Let  dj  be  such  that 
hC«*  is  (y(.la),  l-\)  minimal  left  adjusted.  The  resiainder 
of  the  code  is  constructed  inductive 1/  as  follows. 

Suppose  aethers  {x. ,  have  been  obtained.  Let 

Ojj  ■  If  bCCdj^,  d)  lb)  <  1-1  the  construction  is  com¬ 

pleted;  otherwise,  d^j  is  obtained  as  follows.  Let 

t  •  inf  {d*  -  d^i  there  exists  x  c[a,  b]  such  that  u((d^,  d*)  lx) 
£  1-1,}  Clearly  vCCdjj,  ♦  l  -  c)|x)  <  1-1  for  all  c  >  o  and 
all  xkX.  It  will  be  shown  that  there  exists  xcX  such  that 
hCCdjj,  d^  ♦  A U)  £  1-1.  The  n*lst  element  of  the  code  will  be 
chosen  to  be  {x,^,  ♦  Q}.  If  there  is  more  than  one  x 

such  that  u((dNI  ♦  £)lx)  £  1-1  then  any  such  x  may  be  chosen. 

Let  F(yl  be  the  distribution  function  of  u.  u  «  y  iapiies 
F(y)  is  continuous.  Observe  that  for  any  xcX  FxCy)  ■  F(y-x) 
where  Fx(y)  is  the  distribution  function  of  y(.(x).  Also  if 
xcX,  k  >  0  then  uCCd^  dj^  ♦  k)|x)  ■  Ffd^  -  x  ♦  k)  -  FCdj,  -  x). 

Let  ^  \  l  then  for  each  n  there  exists  x^  such  that 

'Wn  -  *»  ♦  V  -  F(dN  -  V  -  »-*•  <«Xx  C  X  h.nc  i, 
bounded  and  therefore  has  a  limit  point,  say  x' .  Let 
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<\}nw  -*’•  ^  *<**  ‘  «•  -  F(«N  ‘ 

■  lin  (F(d^  -  xfl  ♦  ln  )  -  FCdjj  -  x  )J  *  1-X.  This  proves 

V"  k  k  * 

the  induction  step.  It  is  deer  that  the  code  constructed  above 
has  length  N(S,  X,  U) . 

In  the  preceding  construction  the  specific  nature  of  the 
code  is  not  readily  apparent.  In  the  case  where  f  is  shell 
function  one  can  construct  the  code  so  that  it  is  axire  trans¬ 
parent,  and,  hence  easier  to  manipulate. 

3,22  Construction:  Let  S  be  a  channel  of  type  A.  Let 
f  ■  du/dy  be  a  bell  function  and  let  0  i  X  <  1  be  given.  Let 
X  ■  [a,  b]  and  Y  ■  Cc,  d)  where  both  c  *  -  •  and  d  »  ♦  •  are 
allowed. 

Since  f  is  a  bell  function,  there  exists,  by  theorem  3.11, 
a  connected  set  (tj,  t2l  of  minimal  y-measure  such  that 
wCtj,  t2)  t  1-X.  The  nusfcers  tj  and  t2  will  be  used  to  con¬ 
struct  a  X-code  of  length  N(S,  X,  U). 

Let  (c,  dj)  be  the  (u(.ja),  l-X)-minimal  left  adjusted 
interval  in  (c,  d) .  This  interval  exists  by  lemma  3.17.  The 
remainder  of  the  X-code  is  constructed  as  follows.  Let  the  ith 
pair  be  denoted  by  (**•  (d^j,  d^J.  The  i*lst  pair,  if  it  exists. 
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is  constructed  by 

Cast  1:  If  dL  !  a  ♦  tj  let  (dj,  d.^)  be  th*  (u(.|al,  l-*>- 
minimal  left  adjusted  interval  in  (d^,  d)  and  let  x^j  ■  •• 

Case  2:  If  a  ♦  tj  <  d£  <  b  ♦  tj  let  d i<((1  and  xi4j  •  dL  -  tj. 

By  definition  cf  tj  and  t2,  it  is  clear  that  (dj,  di+J)  is 

{w(#|x.+jX»  l-A)-minimal  left  adjusted  in  (d^,  dX. 

Case  3:  If  d,  2  b  ♦  tx  and  wCd^  d)  [b)  <  l-A  the  construction 
is  coapleted;  otherwise  (d^  i  b  ♦  tj  and  u(£d^,  d)  (b)  2  1-A) 
let  (d.,  di^1)  be  the  {u(.|bX,  l-A}-«dnimal  left  adjusted  inter¬ 
val  in  (d.  ,  d)  and  let  x^  *  b. 

N 

The  code  constructed  above  will  be  labeled  by  ((x^  ^i»i* 

It  is  clear  that  { D.  .  is  ({u(.  Ix.lJ.j,  1-Al-minimal  left 

I 

adjusted  in  Y  and  that  if  {(Si,  j^)  )iml  is  ®»y  ***  of  pairs 

such  that  (Bi^.i  is  (lu(.  1^1  xJ.j.  1-Al-minimal  left  adjusted 

in  Y  then  bN  !  where  Bi  -  (a.,  bj.  Thus  it  follws  from 
theorem  3.18  that  N  «  N(S»  A,  (i).  Any  meaber  of  the  code  con¬ 
structed  above  of  the  form  (a,  will  be  called  an  a-pair; 
any  member  of  the  form  (b,  DjX  will  be  called  a  b-pair. 

It  is  clear  that  the  code  constructed  above  is  not,  in 
general,  the  only  connected  code  of  length  N(S,  A*  UX*  F°r 
reference  later  this  code  will  be  referred  to  as  a  connected 
code_of_t££e_l_. 


From  the  constructions  outlined  above  it  is  clear  that  a 
more  precise  notation  for  the  code  is  {X^CX),  Di^^i«l*  ** 
However  when  there  is  no  possibility  of  confusion  the  X  will  be 
impressed. 

3.23  Definition;  Let  S  ■  {X,  y(.|xl,  (Y,  Y'J}  be  a 

channel  of  type  A.  Let  0  *  X  <  1  and  let  {*i#  be  a 

connected  X-code  of  type  1.  This  code  will  called  full  if 

N 

u((Y  -  II  D.)lx)  ■  0  for  xcX.  The  tern  full  code  will  always 
i-1  1  - 

refer  to  a  connected  code  of  type  1  which  is  full. 

It  is  easy  to  see  that  given  any  N  >  N(S,  0,  X)  there  exists 
XN  such  that  N(S,  XN,  II)  ■  N,  and  the  X^-code  is  full  iff  given 

X  <  XN  then  N(S,  XN#  li)  >  N(S,  X,  U) . 

3.24  Remark;  Lot  S  -  {[a,  b] ,  v( . |x) ,  C*»  Y*)  }  be  a 
channel  of  type  A  with  additive  noise  u.  Let  0  f  X  <  1  be 
given.  Then  there  exists  a  channel  S^,  more  precisely  an  input 
alphabet  [a^,  b^J,  such  that  the  X-code  for  is  full.  In  order 
to  verify  this  statement  it  suffices  to  demonstrate  such  an 
alphabet.  To  remove  any  possibility  of  aabiguity  let  the  family 
of  measures  {u(.|x}|xcR}  be  defined  by  u(A|x)  ■  u(A-x).  for  all 
xcR,  AcY*.  Since  w  <<  Y  there  exists  an  interval  I  «  (t^,  t^} 
such  that  u(I}  ■  1-X  and  given  any  interval  J  withn(J)  <  tj  -  tj 


then  w(J)  <  1-A,  It  is  easy  to  see  that  that*  exists  y^  >  a 
such  that  u(A|x)  <  1-A  for  all  x  -  AC(-  •*,  a  ♦  t2)j 
similarly  there  exists  y2  <  b  such  that  u(A|x)  <  1-A  for  all 
x  i  y2,  AC(b  ♦  t^,  -J.  Let  [a*,  b'J  be  defined  by  a'  ■  a, 
b'  ■  y^  ♦  (b  -  y2)  ♦  2(t2  -  tjX»  and  let  S'  be  the  channel 
with  alphabet  [a*,  b'J.  Let  {x^  be  ***•  t)T?e  1  A-code 

for  S'.  It  is  easy  to  see  that  if  ♦  tj,  b  ♦  t2)  then 

Y(Djl  ■  t2  -  ty  Let  {x!,  Dp^  be  the  code  constructed 
analogously  to  {x^,  op?aj  where  one  uses  right  adjusted  se- 
quences.  Let  a  »  sup  {ylycD^  f\  (-  a  ♦  t^  ^  0  ) ;  let 
8  ■  inf  {ylycD! ,  D!  f*\  (b  ♦  t2,  •)  i  9  }.  Given  C  -  b‘  let  8(C) 
be  the  point,  considering  the  channel  with  input  [a,  C] •  ana¬ 
logous  to  8.  Then  given  C2  -  Cj  ?  b*  then  S(C2)  »  BCCjl  ♦ 

C2  -  Cjt  Thus  one  can  clearly  choose  b*  1  b'  such  that 
6(b*)  -  o  is  an  integral  multiple  of  (t2  -  tj) .  The  type  1 
A-code  for  the  channel  with  input  alphabet  [a,  b *]  is  full. 

In  3.21  it  is  clear  that  for  a  given  0  5  A  <  1  and 
1  5  i  i  N(S,  A {  U}  there  may  exist  several  xeX  such  that 
u(D^[x)  i  1-A,  We  define  X*  »  {x|xcX  w(D^[x)  l  1-A). 

3. 25  Theorem:  Let  S  be  a  channel  of  type  A.  Suppose 
N(S,  A,  U)  -  N(S ,  Aj  for  all  0  5  A  <  1.  Let  0  *  A  <  1  be  given 
such  that  the  A-code  {x^(AJ,  D^(.A)}^j*  is  full.  Tnen 

given  l5ijfj*N(S,A,  U)  and  DjCA),  A^C  D^(A)  such 
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that  thara  axists  x?  c  X*  for  which  pfA^xfX  ■  w(Aj[x|X  i  0 
than  w(A.jxj)  l  w(A^ |xjX  for  all  xj  c  X^. 

Proof: 

Suppoaa  thara  axists  A  such  that  tha  A-coda  is  fhll 
and  thara  axists  A.  C  (  A) ,  CDj(A)  for  sone  i  i  j  such 

that  thara  axists  xf  c  x|,  xj  e  X^  such  that  uCA^xp.  * 
p(Aj|x^J  >  0  and  y(Ai|xJ}  >  w(A^jxJ).  Than  wCCD^^ C^J  *  \l 
UAjXlxp  -  1-A  and  uCCDjCAl  -  A^)  UA.)[x?)  >  1-A.  Hance 
choosing  Dt(A)  ■  D^(A)  -  A.*  \J  A ^  and  Dj(AXC  (Dj(AX*-JA.)  - 
Aj  such  that  u(Dl[xtX  *  1-A  one  can  obtain  a  A-code 

{^i*  **  ^  such  that  there  exists  at  least  one  xcX 

N(S,  A,  11} 

such  that^Y  -  (J  B.  [x)  >  0.  It  follows  by  the  abso- 

i-1  1 

lute  continuity  of  A,  (More  precisely  by  tha  fact  that  tha  end¬ 
point  of  each  is  a  continuous  function  of  A)  that  there 

exists  Aq  <  A  such  that  N(S,  Aq)>N(S,  Ao,  U) ,  But  this  i*  a 
a  contradiction  of  the  hypothesis  that  N(S,  AX  ■  N(5»  A,  U)  for 
all  0  <  A  <  1. 

3,26  Corollary:  If  there  exists  a  full  A-code 
{Xi»  Di)^*  A*  ^  and  for  sons  1  2  i  i  j  5  N(S,  A,  U)  there  is 
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AjC0!*  AjcDj  such  th4t  w(Ai |xp  i  0  i  u(^|x?)  and 

£Cyil*Ji/£Cyilxi)  *  f(yjlxj^ffyjlxP  for  *laMt  a11  [y] 

y .  cA, ,  y. cA.  and  aoaa  xPcX^,  xfcX^  then  there  exists  0  <  X.  <  1 

lijj  iijj  * 

such  that  N(S,  X^)  >  N(S,  X^,  IQ. 

Proof? 

Let  EjCA.,  Bj^Aj  5Uch  that  " 

ti(Bj(x|}  >  0.  Then 

'  f(yU?) 

wCBjlxJ)  -  /  f(y[xj)dy  •  /  f(y(x?)  flyjx,jy  dy  > 

,  .  f(y|*5)  . 

I  I xi^  •rrn^fr dT  ’  l,(Bi|x5) 

3,27  Corollary:  Suppose  f  ■  dy/dy  is  a  bell  function 
and  is  unbounded.  Then  there  exists  a  channel  S  of  type  A  with 
additive  noise  y  and  0  <  X  <  1  such  that  N(S,  X}  >  N(S,  X,  U) , 

Proof: 

It  follows  from  the  definition  <vf  a  bell  function 
that  there  exists  at  most  one  point,  y  ,  such  that  "Tim  f(y)  *  •. 

r*. y0 

(fee  may  as  well  as  suae  that  yQ  ■  0. 
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Observe  that  it  suffices  to  show  that  there  exists  a 
channel  S  ■  {[a,  b],  u(,|x),  (Y,  Y1)}  and  a  full  A-eode 
(*if  D^i.i  *uch that  either  (1)  b^  <  a  <  x2  or  (2)  <  b  <  ^ 

where  bjl.  In  the  first  case  fCy la) /f(y Ixji  is 

unbounded  on  while  this  function  is  bounded  on  a  subset  of 
of  positive  u(.lx2)  measure.  Similarly,  in  the  second  case, 
f(y ib)/f(yixN_p  is  unbounded  on  D^j  while  this  function  is 
bounded  on  a  subset  of  1^  of  positive  u( .  J  xN_ measure. 

If  either  w(-«,  01  ■  0  or  y(0,  <*1*0  then,  for  a  given 
channel  S,  more  precisely  for  a  given  input  alphabet  [a,  b],  it 
is  clear  that  one  can  choose  0  <  X  <  1/2  such  that  the  X-code 
is  full.  Thus  either  (1)  or  (2)  must  hold. 

Suppose  (— • ,  0)  i  0  /  u(0,  «) .  One  may  assume  with  no 
loss  of  generality  that  y(-«,  0)  >  y(0,  -) .  Observe  that  as 
1-X  increases  t^(X)  decreases.  Thus  there  exists  1-X  <  0) 

such  that  t^(Xl  <  a^.  By  3.24  there  exists  a  channel  S  such 
that  the  X-code  is  full.  Since  tj(A)  <  a^  <  a  (1)  must  hold. 

3.28  Remark:  Sufficient  conditions  for  N(S,  X)  »  N(S,  X,  U) 
will  now  be  shown.  Throughout  the  remainder  of  this  chapter  it 
will  be  assumed  that  f  is  bounded  and  f(0)  l  f(yl  for  all  ycY, 

The  importance  of  the  behavior  of  the  ratio  f(yl/fCy-d)  was 
partially  shown  in  3.26  and  3.27  in  the  form  of  necessary 
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conditions.  The  following  theorem,  which  continues  this 
investigation,  will  be  used  to  derive  a  sufficient  condition 
for  N(S,  X)  -  N(S,  X,  U). 

3.29  Definition:  Given  sets  Bj  and  B2  then  Bj  is  less 
than  B2,  Bj  <  Bj,  if  given  any  yjCBj,  Y2t^2  then  yi  <  y2* 

3.30  Tbeorea:  Let  Xj,  xf  X  with  ^  »  Xj*d,  d  >  0.  Let 
0  <  X  <  1.  If  f(y)/fCy-d)  is  a  decreasing  function  of  y  then 
given  any  Aj,  Aj  with  Aj^  Aj  -  0  and  ufA^Xj)  >  1-X,  i  -  1,  2 
the:?e  exists  BJf  B2C  AjUAj  with  Bj  <  B2  and  uCB.Ix^  l  1-X, 
i  ■  1,  2. 

Proof: 

Observe  that  f(y)/f(y-d)  -  du(.  [xjl/duC.  [x^.  Let 
L  -  inf  {ylwfiAjU  A2)  f)  (-«,  y)  |  Xj)  2  1-X}.  Let 
Bj  *  (A1UA2)/1(— ,  5).  Then,  since  y  «  y,  ufBjl  Xjl  »  1-X. 
By  3,5  B j  is  a  set  of  minimal  y(,[x2).  measure  such  that 
uCBjIxj)  i  1-X,  LetB2=  (£,  -)D  (AjU  f^) .  yCBjlx^  ♦  y(B2|x2) 
*  wCAjlxj)  +  m( A^ l x2)  ,  Hence,  since  yfBjlx.^  £  y(Ajlx2), 
uCB2[x2)  l  W C i x2)  i  1-X. 

3.31  Lemma:  Let  0  £  X  <  1.  Suppose  that  given  any 
Xj,  x2eX  with  x j  <  x2  such  that  there  exists  Aj,  A2  with 
Aj/"\  *2  *  ^  and  wCAjlxj)  i  1-X,  i-1,  2  then  there  exists  Bj, 
B2C  AjU  with  Bj  <  B2,  BjH  B2  -  0,  and  uCBjJxjl  >  1-X, 
i-1,  2.  Then  NCS,  X)  »  N(S,  X,  U). 


PTQof: 

Ut  (x. ,  be  a  X-code  of  length  N(S,  X) ,  It 

will  be  assumed  that  the  indexing  it  such  that  i  «  j  implies 

N 

Xj  <  Xj,  By  hypothesis  there  exists  a  code  {x^»  AJJ^j  with 

Ai  ^  A1  and  A1  <  ^N*  til*  ^KU**11*  i* 

a  code  {x^,  AV}^  with  A"  <  ^  and  A”  <  A“  ,  It  is  easy  to 

see  that  repetition  of  this  logic  proves  the  existence  of  a 
code  {x^,  with  <  B2  <  Bi  *or  3  - ■  i  -  N.  It  is  now 

N 

dear  that  there  exists  a  X- code  (x^,  with  C  <  when¬ 

ever  i  <  j. 

Let  aA  •  inf  {y|ycCi>;  bi  ■  sup  (yiycC).  Let  »  (a^  b^J. 

Then  w(D.lx.)  >  uCCjx^  >  1-X  and  D.f)  -  9  if  i  i  j.  Thus, 

N 

{x^,  Di^i«l  **  *  connected  X-code  of  length  N(S,  X).. 

The  following  theorem  is  an  immediate  consequence  of  the 
two  lemmas, 

3,32  Theorem;  Let  S  be  a  channel  of  type  A  with 
f  ■  du/dir  a  bell  function.  If  f  is  log- concave  then  N(S,  X)  ■ 

N(S,  X,  U)  for  all  0  £  X  <  1.  In  particular,  N(S,  X>  »  N(S,  X,  UJ 
for  all  0  S  X  <  1  if  S  has  additive  Gaussian  noise. 
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It  will  new  be  shewn  that  then  exists  channel*  of  typo  A 
with  f  -  du/dy  a  bell  function  for  which  then  exists  ft  <  *  «  j, 
•  uch  that  N(S,  A)  >  N(S,  A,  IQ. 

3.J3  Definition;  Let  F  be  the  set  of  all  functions  of 
the  following  kind.  Given  0  <  $  <  it  l  <  t  <  i/»#  define 

fCy)  *  t  if  y  s(-s/2t  s/2). 

f(y)  .  u  -  ifil  if  y  c(-l/2,  -a/2)  U  C»/2,  1/2). 

•  0  elsewhere. 


Figure  It  A  Typical  Meatier  of  F 


.3^34  Remark ;  Given  feF  then  f  induces  a  probability 
measure,  wft  on  the  Borel  sets  in  a  natural  w ay,  i.e. 

uf(A)  •  /  f(y)dY 
A 

for  all  Aef , 

3^35  Definition.  Given  l  >  0  and  fsF  let  SCf,  £)  denote 


any  channel  {[a,  b],  y(.|x),  (Y,  Y*}>  with  additive  noise 
and  b-a  ■  i. 

It  will  now  be  shewn  that  for  certain  masters  of  the 
family  {S(f#  f)  |  ftF,  l  >  0}  N(S,  X)  •  N(S,  X,  (!>  for  all 
0  i  X  <  1  while  for  other  menbexs  of  the  family  equality  does 
not  always  hold. 

3.36  Remark:  Let  S  be  a  channel  of  type  A.  Given  ycY 

define  f*(yl  ■  {sup  f(y[x)lxeX}.  Let  F(y)  ■  /  f*(y)dr.  Then 

Y 

N 

given  0  2  X  <  1  and  any  X-code  {x^  A.  )?bj  then 
l  MCajjxj)  <  F(Y1. 

ix  JLtraa;  Given  l  >  0,  fcF,  N(S(f,  X)  »  N(S(f#  , 
X,  U)  h.  \  >  1-st. 

Proof ; 

Let  X j  -  1-st  such  that  X^-code  is  full.  Let 

N  *? 

{x.,  D. ,  be  a  full  code.  Then  l  y(D. |x-l  ■  F(Y). 

1  x  l-i  i.j  1  1 

Hence  N(S(f,  l) ,  X^  ■  N(S(f,  £) ,  X^  U\.  Suppose  there  exists 
X2  <  such  that  N(S(f,  t \ ,  X2)  -  N(SCf,  £)  *  Xj)  -  N.  Let 
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ui»  Vi-1  be  *  Vcod#*  T**  i  i  nu-a2x  >  n u-^X 

i*l 

■  F(Y) .  Which  is  a  contradiction.  hence  A  <  A,  implies 
N(S( f ,  4) ,  A)  <  N(S(f,  £),  Aji,  This  coapletes  tha  proof. 


3.38  Remark ;  Let  {x. ,  A.  .  be  a  full  i-code  with 

■I  ii  l*  x  1*1 

A  <  1-st.  Let  k  be  the  nunber  of  a-pairs.  Let  ■  C*£»  h^X 

Observe  that  bi  -  ai  >  s.  Suppose  x^j  +  b..  Then  ia 
an  interval,  s ay  [a^t  «2] „  la  fact,  see  Figure  2,  o^  ■  b^  ♦  s/2 

and  a2  -  Oj  ■  (l-A-st}/u.  If  b^  -  U  -  s/2X<  there  exists 
an  interval  (6,  a-s/2)  such  that  f(yla^l  •  u  for  all 
y  c (8 ,  a-s/21,  see  Figure  3.  Thus  under  these  conditions  it  is 
easy  to  sec,  Figure  4,  that  one  can  choose  such  that 

there  exists  y^cA^  y2cAj,+  1  such  that 

while  f(y2|a)  •  t  and  fCy2  *  u»  It  follows  from  3,25 

that,  under  the  conditions  specified  above,  there  exists 
0  *  A  <  1  such  that  N(S(f.  £)  ,  A)  >  NCSCf,  £},  A,  IQ. 


Figure  2.  Definition  of  the  Interval  Ca^t  a^} 
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3.39  Um»;  If  at  <  1/3  then  exists  0  <  A  <  1  such  that 
N(S(f,  f),  A)  >  N(S(f.  f),  A,  U). 

Proof; 

If  b-a  <  s  let  A  be  such  that  N(S(f»  ti,  Xt  (J)  ■  3 

U 

and  the  X-code  {x.,  A.  is  full.  Let  A^  ■  (a.,  b.).  Observe 

that  bj  <  a  ♦  s/2.  One  can  assuae  that  x2  ■  (a  ♦  b)/2.  Now 

bj  -  (a  -  s/2)  <  s  <  ,  Thus ,  for  this  case,  the  conclusion 

follows  from  3.38. 

Siqipose  b  -  a  i  s.  Let  XQ  •  aax  {A  [A  <  1-st  and  the  A-code 
is  full}.  Let  {x^  be  a  full  XQ-code  where  A^  «  (a^,  bi) 

and  let  k  be  the  nuxber  of  a-pairs.  Then  either  (1)  bj,  <  a  ♦  s/2, 
or  (2)  b^  >  a  ♦  s/2,  or  (3)  b^  ■  a  ♦  s/2.  Observe  bk  2  a  ♦  s/2 

implies  k  2  2  since  b^  i  a  ♦  s/2,  k  *  1,  and  s  <  1/3  iaplies 

b'2  i  a  -  s/2  ♦  s  ♦  *  s  >  a  -  s/2  +  3s j  but,  this  indicates 

that  b2  -  (a  -  s/2)  >  3s  which  says  that  there  exists 

XQ  <  Xj  <  1  -  st  such  that  N(S,  Xj,  IQ  2  N(S,  X^,  IQ  ♦  1  which 

contradicts  the  definition  of  A  . 

o 

If  (1)  is  true  then  b^  -  (a  -  s/2)  <  s  <  and  the  con¬ 
clusion  follows  froa  3.38.  If  (2)  is  true  then  there  exists 
€  e(a,  b)  such  that  CcX*.  Now,  replacing  (x^,  A^)  by  (£,  A^ , 


on*  obtains  a  code  with  k-1  a- pairs  and  fa-  s/2.  Again 
the  conclusion  follows  by  3.38.  If  (3)  is  true  let 

■  sax  { A { A  <  XQ  and  the  X-code  is  full}.  Let  (C^*  be 

a  fUU  Xj»cod*  and  let  n  be  the  nuaber  of  a>pairs,  Let 
A£  ■  (aj,  bp.  Then  either  (1*)  b^  <  a  ♦  s/2  or  (2') 
b^  >  a  ♦  s/2.  The  conclusion  follows  by  the  sane  argunents 
used  above* 

3,40  Theorem:  M(S(i,  4 ) ,  X)  -  N(S(f,  4),  X,  U)  for  all 
0  S  X  < 1  iff  given  X  <  1  -  st  such  that  the  X-code  is  full  then 
either  N(S(r,  4),  X,  U)  .  2  or  X  J  l  ♦  st  ■  y  - 

Proof; 

The  condition  Xi  l  ♦  st  -  y  -  y  is  equivalent  to 
1  "  ""  y^1  *  (— y— ■)  t . 

To  show  that  the  condition  is  m  -essaiy  assure  N(S(f,  4) , 

A,  £1)  ■  N(S(f,  4),  X)  for  all  0  s  X  <  1.  By  3,39  st  >  1/3. 

Let  XQ  ■  max  {x|x  <  1  •  at  and  the  X-code  is  full}.  Lot 
N 

(xit  be  a  full  X-code  and  let  Dj  ■  (a.^  bp,  Suppose 

N(S(f,  4),  X0,  U)  >  2.  Since  1  -  XQ  >  st  2  1/3  2 

wC(»  -  y,  a  -  s/2)  lajpj  >  a  -  s/2.  If  bj  <  a  ♦  s/2  then 

1  -  Xo  -  uC(a  -  y,  bp  |  a)  -  *  (b ;  -  (.  -  f))t  which,  by 

3,38  is  greater  than  or  equal  to  If  b^  2  a+s/2 
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it  is  tuy  to  see  that  -  a^  >  -iji  ♦  %  fbr  1  £  i  £  N.  Thus 

62  *  a  ♦  s/2  ♦  ♦  s;  hence  b2  -  (a  -  s/2)  >  ♦  2s.  It 

follows  by  the  naxinality  of  that  -^2.  <  s.  Hence, 

1  -  A  •  u(Djla)  >  u((a  -  j,  a  ♦  s/2)|a)  -  ♦  st  >  ^  ♦ 

(■i*i)t.  This  the  condition  is  necessary. 

To  show  that  the  condition  is  sufficient  let  0  <  A  $  1  ♦ 

*t  "  T  *  7*  Ut  <  x2  and  let  Aj,  A^Y* ,  A1/r)  ^  •  {I.  If 

>  s  and  *2  -  <  -^2.  then  uCAjxj)  ♦  uCAjlxj)  <  1  -  st  ♦ 

(i^i  ♦  s) t  <  2(1-A).  Thus  if  y(Ai|xi)  *  1-A,  i  -  1,  2  then 
either  (1)  Xj  -  Xj  2  -^2.  or  (2)  x2  -  Xj  <  <  *. 

Case  1;  x2  -  x1  <  -^2.  <5,  It  is  easy  to  see  that 

Xj  +  s/2  >  x2  -  s/2.  Let  f*(y)  -  aax  (fCylxj),  f(y|x2))  and 

let  u#(A)  ■  /  f*(y)dy  for  all  acY1,  Observe  that  given 
A 

x2  -  s/2  <  y  <  xx  ♦  s/2  then  :**(A1(JA2)  ■  uCCAjUA^/K--,  yljxp 
♦  u((A1UA2)/)(y,  -)  Let  C  *  inf  (y| vCCAjU A^fK— .  y)|xj) 

>  1-A.  Let  B2  «  (A1UA2)H(-.  O.  The"  uCB^Xj)  »  1-A.  Ut 
B-,  -  (A^A^^U,  •)•  Cbserve  that  v*£CA1U A2)/)(-"»,  x2  -  s/2)) 

<  1-A,  and  u*((A1UA2)nt«1  ♦  s/2))  <  1-A,  If  follows,  since 
w'CAjUAj)  >  2£1-A),  that  x2  -  s/2  <  L  <  Xj  ♦  s2.  Hence, 

2(1—  AJ  <  u*(A1UA2)  ■  uCBjxj)  ♦  u(B2|x2).  Therefore, 
u(B2|x2)  >  1-A. 
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CM>  2:  xj  -  ^  n>en  U  *  d,c”“i"S 

function  of  y.  Him,  by  S.30,  ther.  exist.  Bj  <  BjC  *,U  *2 
such  that  £  1-A,  i  m  l,  2. 


If  follows  from  3.31  that  N(S(f,  £)  ,  A,  *  N(S(f,  l) »  X) 
for  all  A  5  1  ♦  st  -  y  -  y* 


2 

Suppose  N(S(f,  Q,  X,  0)  *  2  and  the  A-code  (x^  A^.j 

is  full  then  •  a,  %2  ■  b  and  Aj  ■  (a  -  If  b  *  a  ' 

then  u(A1|a)  ♦  yCAjfb)  •  1  ♦  (b-a)t  •  F(y) .  If  folios  from 
3.36  that  NCS(f,  L )  .  A)  -  N(S(f,  l)  ,  A,  U)  .  If  b  -  a  >  s  then 


s 


>  a  ♦  s/2.  Hence  if  A  is  any  set  of  minimal  y- measure  such 

that  y(A)  i  1- A  then  w(A)  -  (l-A-st)/u  -  ^(b+a+ll.  Hence 
N(S(f,  l),  A)  ■  2. 


By  lemma  3.37  N(S(f,  l\ ,  A)  -  N(S(f,  *1 .  A,  U).  for  all 
A  t  1-st.  This  completes  the  proof. 

3.41  Corollary :  Given  l  >  0  there  exists  feF  such  that 
N(S(f,  £).  A)  >  N(S(f,  Q,  A,  10  for  some  0  <  A  <  1. 


Proof: 

Qioose  fe F  such  that  st  <  1/3. 


3,42  Corollary:  Given  feF  with  st  >  1/3  there  exists 
l  >  0  such  that  N(S(f,  l\,  A)  -  NCSCf,  D .  A.  U>  for  all  0  <  A  <  1. 


so 


Proof: 

Choose  t  ■  3s  -  —.  Lot  a  •  a  ♦  s  •  y— ,  8  •  b  -  s  +  y»« 

Then  w((*  -  y,  a)  | a)  ■  u((8,  b*  y)  |b)  •  *t..  6-a  ■  s.  Hone*, 

given  X  <  1  -  st  such  that  the  X-code  is  full  then 
N(S(f,  t\t  X,  U)  -  2. 

3.43  Corollary:  Given  feF  with  1/3  <  s  <  then  exists 

>  0  such  that  N(S(f,  ,  Xi  >  N(S(f,  £  ) ,  X,  £i)  for  some 

0  <  X  <  1  and  there  exisU  y  0  such  that  N(SCf*  *)  * 
N(S(f,  f2),  X,  UJ  for  all  0  <  X  <  1. 

Proof : 

The  second  conclusion  is  an  imnediate  consequence  of 
Corollary  3.42.  To  show  the  first  conclusion  observe  that 
s  <  iaplies  that  1  -  st  >  1  *  st  -  y  -  y.  Not#,  given 

1  -  st  >  X  >  1  ♦  st  -  y  -  y  one  can  dhoose  ^  >  0  such  that  the 
X-code  is  full  and  N(S(f,  l  )t  X,  U)  >  2. 

3.44  Corollary;  N(S(f,  f)  ,  X)  -  N(S(f,  f) ,  X,  U)  for 
all  l  >  0  and  all  0  <  X  <  1  iff  s  > 

4t 

Proof: 

The  necessity  of  the  condi cion  is  proved  in  3.43. 
Suppose  s  -  Then  1  -  st  £  1  ♦  at  -  y  *  y. 


3. 45  Conjecture:  It  has  been  shown  that  if  f  is  a  bell 


function  one  may  still  have  N(S,  X)  >  N(S,  X,  U1  for  soma 
0  -  X  <  I.  The  author  has  been  unable  to  formulate  a  proof  that 
if  N(S,  X>  -  N(S,  X,  li)  for  all  0  <  X  <  1  then  f  must  be  a  bell 
function.  However,  with  the  aid  of  theorem  3.2S  many  non-bell 
functions  have  been  investigated  and  in  all  cases  it  has  been 
possible  to  find  a  X  such  that  N(S,  X)  >  N(S ,  X,  U) .  This,  along 
with  intuitive  feeling,  has  led  tn  the  conjecture  that  N(S,  X)  » 
N(S,  X,  U)  for  all  0  5  X  <  1  implies  f  is  a  bell  function. 
Examples  are  listed  below  of  channels  of  type  A  where  f  is  not 
a  bell  function.  It  will  be  observed  that  in  each  exasple  a 
family  of  functions  (hence,  a  family  of  channelsl  is  defined, 
and,  in  each  example,  given  c  >  o  there  exists  a  member  f  of  the 
family  and  a  b  ill  function  g  such  that  |  f(yl  -  g(yl|  <  e  for  all 
ycY.  Moreover,  N(S(g,  £)  ,  X)  ■  N(Stg,  £)  ,  X,  IQ.  for  all 
0  5  X  <  1. 

3.46  Example;  Given  0  <  8  <  1,  0  <  6  <  1/2.  Let 
f(yl  «  1-8  for  y  e(-  6/2,  6/2) 

-  1  fory  t(-  1/2,  -  5/2)  U  (6/2,  1/2) 

*  0  otherwise 

Let  X  -  [0,  l/2i  let  X  -  1  -  yf(-  1/2,  1/4).  Then 

N(S ,  X,  li)  ■  2  and  tne  X-code  is  full.  Observe  that  f(y[l/2) 

>  f (y l o)  for  all  y  e(0,  5/2)  and  f(y|o)  >  f Cy 1 1/2)  for  all 


y  c 1 1/2  -  6/2,  1/2),  Hence,  by  3.2S  there  exists  0  i  1  '  1 
such  that  N(S,  X)  >  N(G,  X,  U)  . 

Define  g(y)  ■  1  for  y  c  (-  1/2,  1/2) 

■  0  otherwise 

Then,  for  0  <  8  <  e,jf(y)  -  g(y) |  <  e.  It  is  clear  that 
N(S(g,  £),  X)  -  N(S(g,  t) ,  X,  U )  for  all  0  <  X  <  l. 


Figure  5.  The  First  Counter  Example. 


3,47  Example:  Given  feF  with  s  l  Define  to 

* 

,  l*s, 

be  the  family  of  all  function  defined  by:  Given  0  <  m  <  u(— j-J 

for  y  cC-  1/2  ,  -  s/2) 

for  y  e(s/2,  1/2). 
otherwise 


define  g(y)  ■  -my  ♦  u  -  m(i~) 


,  1*5, 

ny  ♦  u  -  tn(-y-) 


f(y). 


Given  X  <  1  -  such  that  there  exists  a  full  \-code 
N 

{x.  ,  D.  };  ,  then  there  exists  1  <  j  *  N  such  that 

1  X  1*1 

a  -  s/2  <  y  <  a  ♦  s/2  for  all  ycD^.  Hence  there  exists 

xtcX*.  -{a}  such  that  [a  -  xtj  <  .  Hence  there  exists 

i  J  j  l 

V ,cD^  such  that  ffyjxt)  >  ftyjx^.  New  fCylxj)  •  f(y  (x^) 

for  all  yeDj .  Thus  by  3,25  NCS,  A)  >  N(S,  A,  IQ  for  sane 
0  <  A  <  1. 

Given  c  >  0,  then  for  0  <  m  <  ti  |f(y)  -  g(y)  |  <  e  for 
all  ycV.  N(S(f,  •£)  ,  A)  -  N(S(f,  t )  ,  A,  U)  for  all  0  <  A  <  1 
by  3.44. 


Figure  6.  The  Second  Counter  Example. 


CHAPTER  IV 


ADMISSIBLE  MEASURE 


It  is  reasonably  clear  that  in  formulating  necessary  and 
sufficient  conditions  for  N(S,  X)  ■  NCS,  X,  U)  one  need  not 
worry  about  every  measure  in  the  family  M  ■  (u(«|x):  xcX},  In 
fact  for  a  fixed  value  of  X,  say  Xq,  if  it  is  known  that  there 

N 

exists  a  connected  AQ-code,  {  (x^ ,  of  length  N(S,  Xq)  , 

then  N(S,  Xo,  U)  ■  N(S,  Xq)  regardless  of  tha  nature  of  the  meas¬ 
ures  in  the  set  {y(,[x):  xc(X  -  {x^)^_j)}.  in  this  chapter  a 
method  will  be  defined  for  delineating  those  measures  which  can¬ 
not  affect  the  value  of  N(S,  X,  U} 

Throughout  this  chapter  both  the  input  and  the  output  will 
be  subsets  of  the  reals;  y  will  be  Lebesgue  measure;  and  U  will 
be  the  usual  topology  for  the  reals. 

4.1  Definition:  A  measure  y(.jx)  is  admissible  if  and 
only  if  there  exists  a  connected  set  AeY'  such  that  y(A[x)  >  0, 
u(Ajx')  *  ufAjxl  for  all  x'cX,  and  there  exists  x"eX  such  that 
y(Ac[x")  >  0. 

4.2  Definition:  Given  0  5  X  <  1  a  measure  y(.[x)  is 
A-admissible  if  and  only  if  there  exists  a  connected  set  AeY' 
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such  that  u(A[x)  *  1-.X,  *(Alx')  f  w(A|x)  for  all  x'cX,  and  there 
•xiata  x”sX  such  that  v(AcjxM)  >0. 

4.3  Theorem:  Let  X*  ■  {xcX:  u(.|x)  is  admissible}:  Let 

X.  ■  (xcX:  uC. lx)  is  X- admissible ) .  Let  (X  }  _  be  a  sequence 

a  n 

m 

such  that  0  i  X  <1  which  converges  to  one.  Then  X*  *  (^J 

n  i*l  n 

Proof : 

Any  X- admissible  measure  is  clearly  admissible.  Hence 

m 

(J  XA  C  X*»  Let  xcX*,  then  there  exists  a  connected  set  AcY* 
n«  1  n 

such  that  u(Alx')  -  u(A|x)  for  all  x'cX,  u(Ac|x")  >  0  for  some 

x"sX,  and  u(Alx)  >  0.  Thus,  there  exists  n  such  that 

u(A[x)  >  1-  X  hence  xcX.  . 

n  n 

4.4  Remark;  If  the  channel  is  sc  mi -continuous ,  then  for 
any  0  2  X  <  1  and  any  connected  X-code  (Xj,  Dp,  , (xn,  Dp, 
there  exists  connected  a  X-code  (xp  Dp,  ...,  (xp  Dp  where 
xJcX^  for  i  •  1,  2,  ...,  n  unless  xx  is  empty  which  iapiies 

N(S,  X,  U)  ■  1.  However,  for  the  general  channel  this  need  not 
hold  since  maximums  may  not  be  attained.  The  following  theorem 
and  corollary  show  that  this  is  true  for  certain  interesting 
special  cases. 


4.5 


Theorem : 


Let  S  •  fX,  u(.jx),  (Y,  Y* j}  be  a  given 


channel;  let  S*  ■  (X*,  u(.ix),  (Y,  Y')}.  Then  if  X  is  compact 
and  w(. lx)  is  continuous  in  x,  i.e.  u(A|xl  is  a  continuous 
function  of  x  for  each  AeY',  then  N(S*,  X,  LQ  »  N(S,  X,  U) 
whenever  N(S,  ,  U)  >  1. 

Proof : 

Suppose  N(S,  X,  UX  >  2.  Let  {x^  D.}^'  ^  be  a 

connected  X-code  of  length  N(S,  X,  U) .  Let  1  5  i  £  N(S,  X,  U) . 
Since  u(Ih[x)  is  a  continuous  function  of  x  and  X  is  compact 
y(Dj[x)  has  a  maximum  at,  say,  x'cX.  for  all  j  j  i ; 

hence,  there  exists  x"cX  such  that  u(D^|x'')  >  0.  Therefore, 
x[eX  .  If  follows  that  N(S\  X,  U)  *  N(S,  X,  U) . 

4.6  Corollary*  c  b?  the  channel  dcfir.adin  4.5.  If 

<<  Y  for  all  xcX  then  N(S*,  X,  U)  -  N(S,  X,  U)  for  all 
0  <  X  <  1. 

Proof : 

Suppose  there  exists  0  <  X  <  1  such  that  N(S,  X,  U)  * 

Given  x  cX  then,  r-incc  u  «  y .  there  exists  a  connected  set  DeY’ 
such  that  u(D|xo)  *  1-X.  Hence  utDC[xQ)  =  X  >  0.  Let  x’eX  be 
such  that  u(Uix)  has  a  maximum  at  x*.  Then  x’eX^. 


The  following  •sample  demonstrates  a  channel  with  an 
uncountable  input  alphabet,  in  fact  a  closed  interval,  where 
the  set  of  admissible  measures  is  countable. 

4,7  Example:  Let  X  ■  [o,  1]  and  1st  (Y,  Y*1  be  the  real 
Rtsnbers  ar.d  the  borel  sets  respectively.  Define  measures 
and  by 

w,(A)  -  /  e‘y2/2  dy, 

sm  a 

W2(A)  “  /  e’y?/8 

2/jr  a 

for  each  AcY'.  Let  X2  -  (x|xeX,  x  is  irrational);  let  Xj  »  X-X2. 
The  family  of  measures  {w(.|x)|xcX>  is  defined  by 

y(A.|x)'  -  tjCA-.r)  if  xtX1.-{o)  -  {1} 

u(A|x)  ■  u2(A-x)  if  xeX2  U  {o)  U(l) 
for  each  AcY* . 

It  ia  easy  to  see  that  X^CX*.  In  fact,  if  x  ■  0  let 
A  ■  (-«,  o) ;  if  x  *  1  let  A  »  (o,  «•) ;  if  xeX^Co,  11  let 
A  -  (x-1,  x*l).  Then  y(Alx)  >  0,  uCAC|x)  >  0  and 
y(A[x)  2  u(A|x')  for  all  x'cX. 

Consider  x^cX2 •  Let  AcY' ,  A  connected, with  uCA|xl  >  0 
and  u(Ac|x'l  >  0  for  some  x'cX.  Then  there  exists  a,  8cR 


with  j  ■  (x-a,  x*31  and  either  | x- a ?  <  -  or  [x»a|  «  •.  If 
•ith*r  x-a  t  (o,  1)  or  x*6  c  (o,  U  there  exists  x^cX^Co,  11 
such  that  u(A[x^j  >  vCAjx^).  If  x*6  •?  0  then  u(A|0}  >  uCA^l 
and  if  x-a  £  1  then  u ( A 1 1)  >  uCAjx^l.  It  follows  that  x^X*. 
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CHAPTER  V 

SUFFICIENT  CONDITIONS  FOR  FINITE  CAP *r TTY 

Since  many  of  the  studies  in  information  theory  involve  the 
channel  capacity,  it  is  highly  desirable  to  knew  when,  the  capac¬ 
ity  is  finite.  In  this  chapter,  sufficient  conditions  are 
obtained  for  a  channel  with  additive  noise  to  have  finite  capac¬ 
ity.  It  will  be  assumed  that  the  input  and  output  are  subsets 
of  the  real  numbers. 

5,1  Lemma:  Let  M  *  (y(.|x)|x£X  be  a  family  of  probabi  lity 
measures  defined  on  a  measure  space  (Y,  Y').  Suppose  there 
exists  a  probability  measure  y  such  that  u(.!*l  <<  x  for  all 
xcX  and  { du ( .  [ x ) /dy  |xcX }  is  uniformly  bounded.  Civen  n  >  o  let 

yn(,|u)  and  yn  represent  the  product  measures  an  tv11#  lYnl'). 

Then  giver,  r  >  o  there  exists  6  >  o  such  that  given  ucXn  and 
Ac(Yn)'  then  yn(A|u)  -  c  implies  Yn(A)  -  6°. 

Proof : 

Let  f(y|x)  ■  du(.|x)/dv.  Suppose  {f(y|x)!xcX}  is 
uniformly  bounded  by  M  i  1,  Let  c  >  o  be  given.  Let  6  •  c/M. 
Given  ucXn  let  f^Cylu)  =  du  n( .  I  u)/dyn .  Then  ^(ylu)  £  bf*. 

Hence  if  un(AjuJ  T  c  then  y^A)  i  i/bP  2  (c/M)n  «  6n. 


The  preceding  lemma  enables  one  to  determine  N(Sn,  >1  for 
any  fixed  values  of  n  and  1  >  0  provided  that  there  exists  a 
probability  measure  y  with  respect  to  which  the  family 
{du(.  [x)/dy}  is  iniformly  bounded. 

5.2  Thecren:  Let  S  ■  {X,  CY ,  Y'j}be  a  channel. 

Suppose  there  exists  a  probability  measured  y  defined  on  CV,  Y') 
such  that  the  family  {dy( .  [ x)  /dy  |xcX }  is  uniformly  bounded. 

Then,  given  0  <  \  <  1,  there  exists  6^  >  0  such  that  N(Sn,  * 

— for  any  n  >  0. 

,n  ' 

6i 

Proof: 

Let  0  <  i  <  1  be  given.  By  the  lemma,  there  exists  a 
6^  such  that  for  any  n  '  0  and  any  utXn  uniA^  ju)  2  1-X  implies 
yn(A^l  i  o".  Since  is  a  probability  measure,  it  is  clear  that 

there  are  at  most  — —  disjoint  subsets  of  Yn  of  yn  measure  d". 

f  n  a 

The  conclusion  is  now  clear. 

The  following  result,  due  to  Kenperman  [5],  will  be  used 
to  show  that  the  channel  defined  in  theorem  5.2  has  finite 
capacity. 

5.3  Tiiwurem.  Let  S  be  a  channel  with  capacity  C.  For 


0  <  i  <  1  define 
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Then  for  each  0  <  A  <  1 
C  i  C  (A). 

5.4  Corollary:  Let  C  be  tha  capacity  of  the  channel 
described  in  theorem  5.?.  Then  C  <  *», 

Proof : 

Lat  0  <  A  <  1  be  given.  By  theorem  5.2,  there 

axists  5^  >  L  such  that  N(Sn,  A)  5  — i— .  Hence, 

6A 

c  *  c(A)  5  inri  log  c~)  -  -  log  tx  <  - 

n~  n 

5.5  Theorem:  Let  S  *  {X,  y(,|x),  (Y,  Y')}  be  a  channel 
with  additi»e  noise  u  which  is  absolutely  continuous  with 
respect  to  Lebesgue  measure  y.  If  there  exists  a  choice  for 
the  Radon-Nikodyn  derivative  f  ■  dy/dy  such  that  /  gCyJdy  <  • 
where  g(y)  «  sup  { f(y | x) |xcX }  then  the  capacity  of  S  i*  finite. 

Proof ; 

Suppose  there  exists  a  choice  for  dy/dy,  say  f, 

such  that  /  g(y)dy  <  «,  A  totally  finite  measure,  y’, 

is  defined  on  [Y,  Y')  by  y*(A)  *  }  g(y)dy  for  all  AcY' . 

A 

Let  h(y(x)  ■  dy(.jx)/dy’.  Then,  if  dy'/dy  +  0, 

h(ylx)  •  -  l'  HeRce  {b(y[x)|xcX}  is 

uniformly  bounded  almost  everywhere  [y'J. 
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Define  *(A)  -  Y’(A)/y''vY)  f„r  all  AtY'.  Then  *  is  a 
probability  measure  and  it  is  clear  that  {dy(.  [x)/d\i[xeX}  is 
uniformly  bounded  a.e.  [^].  Thus  by  5.4,  the  capacity  of  S 
is  finite. 

5.6  Corollary:  If  S  is  a  channel  of  type  A  and  f  is 
a  bounded  bell  function  then  the  capacity  of  S  is  finite. 

In  particular,  if  S  has  additive  Gaussian  Noise  the 
capacity  of  S  is  finite. 

The  hypothesis  of  theorem  5,5  requires  that  there  must 
exist  a  choice  for  dy/dy  which  is  bounded.  The  foil  wing 
example  provides  a  partial  justification  for  this  restriction, 

5.7  Example;  Let  S  be  the  channel  of  type  A  defined 

by;  X  ■  [0,1],  Y  is  the  real  numbers,  Y1  is  the  Borel  sets, 

and,  given  AcY'  y(A)  «  j  ■  '  *** - where  A.  ■  An[0,  e-1]. 

A.  2x(-ln  x) 

It  is  clear  that  y  <<  y.  In  fact  f(x)  »  ■—  * - 777“  almost 

2x(-  In  x)*'* 

everywhere  [y]  where  f(x)  is  any  choice  for  dy/dy.  It  is  easy 

to  see  that  lim  f(x)  »  «.  Let  F(x)  be  the  distribution  func- 

x-*o 

function  of  y.  Then 

F(x)  -  0  if  x  *  0 

1  .i 

"/runr  if  0  <  x  - e 

•  1  if  x  i  e* 1 
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-cTTj2 

Hence,  given  0<X<l,Y(A)»ev  ’  for  my  set  A  of 

minimal  Y-m»»sure  such  that  u(A)  »  1-X .  It  follows,  since 

fTTT)2 

X  -  [0,  1J,  that  N(S,  I)  i  «lt  J  .  Therefore  lie  (1-Al  log 

A-i 

N(S,  X)  *  lie  yiv>  *  It  follows  from  Fano's  theorem  that 
X-i  1 

the  edacity  of  S  is  infinite. 

S , 8  Remark :  Let  S  ■  {X,  u(.|x),  (Y,  Y* )}  be  a  channel. 

n 

If  X  can  be  written  in  the  form  X  ■  l  X.  ,  X.  X.  ■  0  for 

i»l  1  x  3 

i  i  j,  such  that  the  hypothesis  of  theorem  S.S  is  true  for 
each  of  the  subchannels  ■  {X^,  w(.|x),  (Y,  Y*).},  then  the 
capacity  of  C  is  finite.  In  theorem  5.5  it  is  shown  that 
for  each  i  there  exists  a  probability  measure  and 
0  <  nu  <  •  such  that  du(.|x)/d*^  f  m^  for  all  xeX^.  Let 

1  n 

♦  "  —  I  i i- *  Then  i|i  is  a  probability  measure  and 
n  i-1  1 

dv(.[x)/d4'  -  n  max  {m^|l  f  i  *  n}  for  all  xeX. 
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